I often need to use a script to parse CSVs (datafeeds, database exports, or just spreadsheets of data) and grab data, or reformat the CSV, but a problem I’ve run into many times is the fact the Excel exports to CSV and does not allow me to enclose the fields in quotes, double-quotes, or…anything. And it separate the fields by commas. And that’s just how it is. (No options!)

So, instead of doing regex acrobatics or parsing contortions, there’s a way to make Excel export in a much easier-to-parse format.

To do this, you must change a Windows XP setting. Click Start->Settings->Control Panel->Regional and Language Options. Next to the Language, click “Customize”. Next, chnage the “List Separator” (under the Numbers tab) to something very unique. I use a tilde (“~”) symbol, due to the fact that I rarely find that within the data I’m working with. But you can use something else such as a pipe symbol.

Then, export your data to CSV. Your fields will have a more unique separator which you can easily distinguish. If you want, you can easily do a search-and-replace of that symbol as well. It’s saved me a lot of time!

So here are nine “tips” – short but helpful things that can make your life easier when you develop a web app, website, or script. I will list these in no particular order – hopefully at least on thing in this list will be of use.

Create a wireframe, sitemap, or content matrix before you ever start coding. Unless you’re hacking something out really quick – plan your programs. Seriously. I see many people just sit down and start coding, but in my opinion they waste far more time doing things that way. It’s far better to plan out your program first, work out the flow of events and logic, and then start coding. And you don’t need to use Omnigraffle or any tools to do this: I map out database structures, navigations, and wireframes on yellow legal pads. So plan your projects in the way that works for you best – but do it.

Use a good text editor or IDE. I personally prefer Crimson Editor. If you’re on a Mac, Textmate is supposed to be good. But I would not recommend trying to code in Dreamweaver as I’ve seen some people attempt. You really need to get into the code – and I recommend this for HTML, CSS, as well as scripting.

Group all “like” files. Have a directory for images, one for CSS, etc. Don’t throw it all in your main directory and expect to keep things in order. Grouping files that belong together will make maintenance much simpler.

Name all include files with a leading underscore. Someone wrote this on their blog (I don’t remember who) and it’s helped me keep things organized for some time. I often have a file such as “__header.php”, “__footer.php”, or “__nav.php” and I can always tell which files are includes for other “main” files or template files by merely glancing at them. Also, by using the underscore character to name them, they will appear at the top of my directory listing by default.

Indent your code with tabs, instead of spaces. It keeps the structure of your code consistent, and it takes less characters to make one tab than to make four spaces. I know it might seem insignificant, but I like to make my files as small as possible.

Use comments, “stubs” and pseudocode when developing – liberally. There’s no reason that you can’t delete/replace this stuff later if you don’t need it. But initially, I find it useful to “jump right in” and actually create the file(s) for my program and make myself notes where things are missing or where I haven’t yet implemented something.

Don’t use inline styles. This is a general rule that I’ve found should rarely be broken. Learn how to control all your pages with a stylesheet. You’ll see the beauty and simplicity of it if you’ve ever had to hunt through multiple pages for stray inline styles when you needed to make a change.

When you begin to get unproductive, go for a walk. I find that there are three activities where I can really think creatively and productively: mowing the lawn, taking a shower, and going for a walk. Seriously, many of my best ideas come to me in the shower! But taking a walk usually clears my head, as well. The essential ingredient here is the “zoning out” factor paired with no interruptions.

If you can’t think of a way to solve a problem, present the situation to a non-programmer. This might sound like it’s the wrong way to approach a coding problem, but it works very well. (Of course, you have to strip the problem down to its essentials and present it in non-technical terms.) But if you have someone you can bounce problems off of and they have no pre-conceived ideas about how you’re “supposed to do it”, you might be surprised at the insights you’ll get from them.

There you have it: nine very specific ideas to help programmers. Whether you code in Perl, Ruby, or PHP or if you’re making small web sites or huge web applications one of these ideas is sure to crop up in the near future. I hope you find at least one of these tips helpful when that time comes!

1. You begin to miss your family, because you’re always doing “computer stuff” they don’t understand
2. You have a lot of accounts at different merchants/affiliate networks and you log into those accounts all the time to check your income
3. You should be making some money

The number 3 part is what keeps people like you and I going. Otherwise we’d have better things to do with our time, right? Right. In fact – even while making money, we have better things to do with our time.

Hmmm…so what can we do about that? How can we maximize our time? I know: let’s do what I’m constantly harping about, and make a computer do the job we don’t need to do. In this case, let’s make a computer log into our affiliate accounts and check stats. Sound good? Good.

For this example, I’m going to show you how to use PHP and cURL to check your Shareasale stats. I thought, “Why not prepare everyone with a nice way to automatically gather their stats?”. But the following lesson can be applied to just about any affiliate site…or any website for that matter.

I assume you are familiar with running PHP scripts, at least. If you aren’t, there are some great books with animals on the cover to show you that. (And, perfectly good free websites to help you learn PHP, as well.)

So let’s just jump into it. Here’s a script that logs into Shareasale, grabs the current amount of income, and shows it to you.

<?php
$username="superoptimist";
$password="ibelieveinmyself";
function return_between($string, $start, $stop, $type)
{
$temp = split_string($string, $start, AFTER, $type);
return split_string($temp, $stop, BEFORE, $type);
}
$curl = curl_init();
curl_setopt($curl, CURLOPT_REFERER, "http://www.shareasale.com/a-login.cfm");
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
curl_setopt($curl, CURLOPT_COOKIEFILE, '/public_html/cookies/cookie.txt');
curl_setopt($curl, CURLOPT_COOKIEJAR, '/public_html/cookies/cookie.txt');
curl_setopt($curl, CURLOPT_POSTFIELDS, "step2=True&username=$username&password=$password&submit.x=40&submit.y=19");
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_URL, "http://www.shareasale.com/a-login.cfm");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$page = curl_exec($curl);
echo "logged in";
curl_setopt($curl, CURLOPT_GET, 1);
curl_setopt($curl, CURLOPT_URL, "http://www.shareasale.com/a-main.cfm");
curl_setopt($curl, CURLOPT_REFERER, "http://shareasale.com/a-login.cfm");
$page = curl_exec($curl);
$total=return_between($page, '<font color=gold> $', '', EXCL);
echo 'got income amount';
echo 'Total Amount $'.$total;
curl_close($curl);
?>


And that’s it. Put your real SAS login info up in there, run it, and you will see a number appear on your screen – like $0.00, if you just opened your Shareasale account. But hopefully, eventually you’ll see some real income.

So why not just log in, manually? Well the point of me sharing this script is not to give you a marginally-useful script that you can use to get your current income. You can do so much more with this script once you think about the possibilities. Here are some:

* Make an affiliate dashboard, and include all your affiliate reports in a nice, single menu with that day’s sales, that day’s hits, current income, etc.
* Grab all your affiliates’ daily sales (running daily via cron) and throw that into a database. Use that to generate reports to show you things like: overall most profitable day of the week, most profitable merchant, product, etc. If you run a few scripts like this – say, one for SAS, one for CJ, one for CCBill, etc. and put it all together you can view reports like this across all your merchants and affiliate networks.
* Grab datafeeds daily or weekly from your merchants, and then update your databases (the ones that run your sites) with the latest products.

Get the idea? This is just an example of basic principles – but cURL is a powerful tool that can almost precisely behave like a web browser – just faster and automatically.

Hopefully you can follow the gist of what this script does, but here are a couple of pointers in case you need some cURL help.

Send what is expected. You must always send GET when the server expects a GET request, and a POST when POST is what’s expected. How to find that stuff out? LiveHTTPHeaders. Possibly the best FireFox plugin out there. At least to me. But there’s probably some weirdo out there with an Fellow Furry Finder plugin or something who thinks that’s the best. (Sorry, that’s not an actual plugin.)

Use LiveHTTPHeaders by selecting it, and then logging into the site you wish to write a cURL script for, and then examining the confusing stuff in the little window. All the stuff you will see is the conversation between your browser and the server. Your goal (in writing an automation script) is to replicate that conversation, or the format of that conversation.

Send in the order that is expected. This means that you must always have the proper referrer with each target URL. If every time a user who goes to accountadmin.php must have come from mainmenu.php, you better put mainmenu.php as your referrer when telling cURL to go to accountadmin.php. Follow the path that you would normally take (as a human, navigating a site).

Respect server resources. Do not do things like hammer a server every five seconds, or visit every hour and downloading a bunch of pics, or anything else like that. Remember Eli’s QUIT tool? Yeah, me too. I miss it. (*Sniff*). See, he shut down the thing just because of someone out there abusing the tool. If you do things like that to other people, they too will probably shut it down (at least, to your IP).

So did we learn something, kids? I hope so. I ain’t here to give ya’ll step-by-step instruction or nuthin’ – but I wanted to at least whet your appetite for some of the things you can do with cURL and maybe give you some ideas. Any questions? I’ll do my best to answer.