Expiring data and paying for data usage

When the internet was introduced in South Africa in the mid 1990s the concept of paying for internet usage was born. Dial-up modems were the standard for most home users. You could stay connected to the internet for the cost of a phone call and a monthly subscription fee. With the introduction of a R 7.00 maximum call charge after 7PM and on weekends using the internet at home for long periods of time finally became a reality. At least for me!

The alternative was to install a DigiNet line, which was super expensive. Just below that on the price scale was a dedicated phone line, but try as I might I was never able to order one. Believe me, I tried! Then came ISDN at slightly more expensive than a normal telephone line it made connecting to the internet permanently at a fixed cost possible for the average home user.

Throughout this evolution, frustratingly slow as it was, the idea of selling the same bandwidth to multiple users started to become a problem. Internet Service Providers (ISPs) would install expensive DigiNet lines and set up points of presence for users to connect to. They would then dump thousands of their customers into one "pool" and leave them to fight it out for the same bandwidth. The ISP has a connection speed to their upstream provider of 1 or 2 megabytes per second, while these pools of users could number hundreds even thousands of individuals. Those with normal dial-ups each contributed 56 kilobits per second, while ISDN users would each account for 64 kilobytes per second, but the ISDN users would be online all the time.

Can you see the problem here? Just close to 160 dial-up modem users users would be able to use all the available bandwidth on a 1 MB upstream connection at an ISP:

56 kilobits per second x 160 users = 1.120 Megabytes per second

As their clients grew ISPs seldom addressed this problem effectively and as a user the best tactic to get consistent connectivity at the advertised speed was to join a new ISP. You would be one of only a dozen or so users for the first couple of weeks, but as the number of clients competing for the same bandwidth grew, the speed went away.

This problem became even worse as ISPs introduced the new ADSL product. ISPs could not balance the price of their expensive upstream connectivity with the price home users could afford to spend. Their solution was simple: Make people pay for the data they use!

Like cars on a highway bandwidth refers to the number of data packets, similar to the cars, that can be accommodated on a given connection, the highway. Just like a rush hour traffic jam, packets are forced to wait, or are simply dropped if the connection can not accommodate their transmission.

Selling the same bandwidth to too many customers now became even more lucrative since the ISPs could now charge a "toll fee" for the use of bandwidth making their expensive data infrastructure exponentially more profitable.

Enter the concept of shaped and unshaped internet connectivity. ISPs would guarantee bandwidth in the form of an unshaped account, like reserving lanes on a highway, for users who could afford a premium rate. The rest of their clients are all left to jostle for the same available bandwidth - shaped accounts.

I would often download large files such as a backup file of a client's web site, for example. With hundreds, or even thousands of people using the same connection at the ISP packets get delayed and dropped, like the traffic jam example above. The large file download times out and fails. The download can be restarted, if possible, but could cause the file to be corrupted, so the best way forward is to restart the download late at night when few other users are active. This process could be repeated several times before the file is finally downloaded and ready to use.

Thus I have now paid several times to transfer the same data between the same two machines! ISP 1 - IGotAFrica 0. My cap is exhausted and I have to top it up. Somewhere along the line the ISPs decided that not only do I pay to transfer data over a fixed connection, I also have a limited time to do so!

So we end up in 2017 and ISPs are now finally being brought to book for this nefarious business model, or so we hope. It is understandable that they are fighting back. Being able to force this pricing model into the market and buying up companies who offer fixed cost products has worked very well. The ISPs tell us that the "transport of data needs to be provisioned" and so the expiry of usage data is justifiable.

In what universe?! Apart from the fact that the infrastructure ISPs put in place is capable of carrying a very small percentage of the bandwidth they sell, the concept of paying to be able to transfer data using a connection I am already paying for is ridiculous. It is not like a highway that needs constant maintenance and upgrades. The router and the equipment I use to connect to the ISP belongs to me, the line I use is paid for via monthly rental. All I need the ISP to do is connect a wire, or cable, or whatever from my connection to their network on to the internet! That's it.

Data that expires? That is fraud! If I sell someone a product and then never give it to them I can go to jail! It's not like my failure to use the data will stop them from shoving yet more customers onto their network! They don't stop their marketing when they have enough users to use all their available bandwidth. The data I paid for magically disappears, but the money I paid remains in their bank account!

All data packages should be uncapped. Throttle my speed, I don't mind that. Make me pay for dedicated, or shaped bandwidth exclusively available to me, no problem. I'll pay more to get a faster connection. That makes sense.

And now for the real kicker: The cache server at your local ISP.

Cache servers are very useful tools. They are used to optimise bandwidth usage in large organisations. When you opened this page all the images you see, the HTML, CSS and Java script files used to display this page were downloaded form our web server to your device and the data used was deducted form your usage account at your ISP. Your device also has a cache. All these files are stored on your device. This page has an expiry date on it which tells your device how often the page is likely to change. When you refresh the page, or you go somewhere else and then open this page again most of what you see is loaded from your cache. That is if the page has not expired and if your cache was not filled by other pages as you browse through the wild, wild web.

In large organisations cache servers are used, as I said, to optimise bandwidth use by saving all the files used to display all the pages accessed by everyone in one office. When I open this page and my colleague in the same office opens this page my colleague will have the benefit of a faster load time, because the files are all served to their device by the cache server in our office. The cache server checks if the page has been stored in its cache and if the page has not expired will serve it up to my colleague.

ISPs use this technology to optimise their billing. They bill your account for data used when you enter their network. Behind the billing software sits their cache servers. When you open this page you will pay to have all the data downloaded from our web server to your device. When another customer of your ISP opens this page they will pay for the data downloaded from the ISPs cache server to their device. The ISP goes one step further. They ignore the expiry of this page. They forcefully cache as much data as they can and serve it up, as old as it might be, to you and all their other customers, from within their own network and then bill you as if the data was downloaded from the site's web server on the internet.

This forces people like me who build web sites to use "cache-busting" techniques to make sure that pages you access are the real latest version of that page. If we did not do this you would open your internet banking site and see someone else's bank details. Also note the "this is not me" facility on some sites. They know you might be looking at a page from a cache server and not the page you have just paid to open.

I would have no problem with ISPs using cache servers on uncapped connections. They would be crazy not to. Charging users for "internet usage" to download the same content form their own network? That is criminal.

Comments