Download Monitoring

Hi,


I have moved in with my new friends to a flat. We often end up using your download limits and have to pay to airtel for extra bandwidth. This often results bad understanding when it comes during the month end and our bill arrives.


I need a software which will monitor the dowload usage which each device that connects to the modem through wifi does. Is there any software or incase any other modem which will work with Airtel connections and comes with a inbuit options to do it. At the max i will need to monitor 15 devices (the phones, tab and laptops).


It enough if the i get the total usage made by each and every device.


Narendran Kishore

Your problem doing it at the device end (apart from collating all the results together) is that you have a mix of device types and things like smartphones won't necessarily have much in the way of options.

You would need to find a router that is able to keep track or use a PC as a gateway device with software on it (but that probably means building a dedicated Linux device probably with 2 network cards).

Afraid I don't have experience of routers that would do this. Are you in the UK or is Airtel an non-UK supplier? I am assuming you are not from UK. And is Airtel a mobile network in which case the router requirements could be different?

Post deleted by David_W

In reply to a post by David_W:

In reply to a post by ian72:

You would need to find a router that is able to keep track or use a PC as a gateway device with software on it (but that probably means building a dedicated Linux device probably with 2 network cards).

The option I use is the FreeBSD based pfSense, which is free of charge. You don't necessarily need two network cards - one VLAN capable network card (ideally an Intel server grade card, though lesser cards will do, depending on your requirements) and a VLAN capable network switch is more flexible. The pfSense forums contain lots of wisdom on the various options.

So far as monitoring traffic per device goes, pfSense supports Netflow via the pfflowd package, which you can use with external Netflow monitoring software. As an alternative, the pfSense ntop package may do all that you need on the pfSense box.

If you have a PC with a couple of network interfaces, you can experiment with pfSense for free, though you will need a spare hard disk partition to experiment with packages, as you cannot use packages when running from the live CD.

I'm sorry, I may be thick, but I do have a First Class Honours in Electrical/Electronic Engineering from Bristol University back in 1970, (the only one awarded that year), but the above post is complete mumbo jumbo to me, and, I suspect, others. Can you explain in more detail?

It's a router/firewall that sits between your network and the internet, and can measure data flow besides other things.

Post deleted by David_W

I'm sorry, but I have to agree with Stevenage_Neil. Your discussions are too heavily overcomplcated and are probably not much relevance to him. Altho' highly computer literate, I would not at all feel comfortable trying to stick some intermediate gubbins in my network just for monitoring purposes.

In reply to a post by XRaySpeX:

I'm sorry, but I have to agree with Stevenage_Neil. Your discussions are too heavily overcomplcated and are probably not much relevance to him. Altho' highly computer literate, I would not at all feel comfortable trying to stick some intermediate gubbins in my network just for monitoring purposes.

The "annoying" thing to me is that I worked for Xerox for 27yrs and was actively engaged in the development of Ethernet, (and variants thereof)..........all these modern acronyms, to those of us not in the know, are confusing! KISS

OK, then - over to you. I've tidied up by deleting the answers you rejected.

Yeah, a technical post in "Technical Discussion". Whatever next?

Please don't misunderstand me. I'm not rejecting anything, but just requesting that you simplify things and talk to the level of your audience and not seem to be lecturing them.

I do find that most of your posts are highly informative and erudite, particularly on the legal aspects, but just a tad too long.

This is just my personal opinion; you are not bound by it.

My exasperation is that I don't see what more I could have done with my earlier replies. What is being asked for is, as I'm about to outline, not a feature typically found on consumer level kit, which means either using some enterprise level kit, or using open source. If I've missed a straightforward solution, I've done so in good faith.

I'm not in a position to write a detailed technical report in user-friendly language, accompanied by a 'ready to go' solution, free of charge. Instead, I offered some pointers in the hope it prompted further research.

If I knew of a straightforward consumer orientated solution, I would have given that solution in a simple and concise reply. As the only answers I could offer were more to the enterprise level solutions, a much higher level of technical knowledge became necessary, though I tried to provide pointers to further information whilst avoiding unnecessary detail.


My initial reply was very terse, but pointed to an open source solution that deals with much of the complexity for you, and which can be experimented with free of charge using virtualisation. Many advanced users deploy this software successfully as their home router, and there is a comprehensive support ecosystem around it (forums / mailing lists / book / commercial support).

After I was asked for some more pointers, I gave a glossary with hyperlinks to relevant information, and an outline deployment scenario for one technique I'd mentioned (which, in practice, often saves money and is widely discussed on the forums for the software in question). I was then criticised for complexity, lecturing and length. If anything came across as a lecture, it was only because I was trying to explain a fairly advanced technique that is quite simple once you get your head around it.


So far as I'm aware, the alternatives to acquiring the knowledge to deploy open source yourself are to pay someone to deploy it for you, or to buy an off-the-shelf commercial product. I'd expect both these options to cost far more than any home user could justify paying.


Here is a very high level outline of possible solutions. It's far more abstract than my previous and now deleted answer, but attempts to explain the issues and give some pointers for further research. The technical level is very much higher than I'd like, but Google, Wikipedia and the like should provide some helpful exemplification.

• Use SNMP monitoring software such as MRTG or PRTG. Even if the existing router supports SNMP, this will only allow monitoring of the total traffic, not traffic per client, using the standard MIBs. If the existing router does support SNMP, it might be worth using SNMP walking software (such as Getif on Windows, or snmpwalk on a Linux/*BSD box with net-snmp installed) to review the available information, especially if the router manufacturer publishes their own MIB. I don't think this solution is viable.
• Install monitoring software on each client. This does not give you a per client breakdown in one place, and requires willing compliance. If compliance cannot be assured, users will just disable the monitoring or delete the collected data from their devices. The heterogeneous client population adds to the complexity of this approach. I don't think this solution is viable.
• Use NetFlow or a similar technique (alternatives: sFlow, Ntop, other software that takes a raw feed and produces statistics). This will do the job well, but you've got to derive the data feed from somewhere. The alternatives here are direct derivation from a replacement router/firewall, interposing a bridge, mirroring the WAN traffic using a suitable switch, or using an enterprise level switch that provides a NetFlow or sFlow feed (the latter arguably being the best option but is obscenely expensive). The first two of these approaches are possible using the the approach I was thinking of in my earlier replies: pfSense on a fanless Intel Atom box with single Intel NIC connected to an 8 port fanless smart switch. Such a system is configured almost entirely via a web based GUI that includes automatic download and installation of pre-prepared packages for NetFlow or Ntop.
• Use a packet inspection appliance. Likely to be available off the shelf (though may require some configuration), but very likely to be obscenely expensive and therefore unviable.
• Use a Wi-Fi access point with WPA2-Enterprise and RADIUS accounting. This is arguably the neatest solution of all, but RADIUS accounting support (as opposed to RADIUS authentication support) typically requires an expensive enterprise level access point, such as HP MSM series or Cisco Aironet. You also have the problem of providing the RADIUS back end - it certainly can be done with FreeRADIUS, which I'm very familiar with, but will require some configuration and testing before relying on it to record data and enforce limits. If there are any wired clients, you'd need to use 802.1X authentication and a switch that provides RADIUS accounting data (most don't, though you might find this in the more expensive HP Procurve ranges or similar from other enterprise manufacturers).
• Use a captive portal with RADIUS accounting, such as the one built into pfSense. This means each client needs to log on to the network before use, as you do on a public hotspot. The RADIUS back-end issues are the same as for the previous solution.
• Switch to an unmetered tariff. This is a straightforward solution, but may not be available at an affordable price on the ISP in question, which would make it unviable.

I took it as read that the solutions requiring enterprise level hardware were out of the question on budgetary grounds, though it's possible you'll find older devices on eBay. The most likely candidates for enterprise level Wi-Fi equipment are older 802.11a/b/g (or b/g) enterprise level access points that are being discarded that has been discarded in favour of newer and faster 802.11n or draft 802.11ac equipment.

If you want to try the Wi-Fi route with a limited budget, 3Com 8760 supports RADIUS accounting, has simultaneous dual band 2.4GHz and 5GHz support (which helps in areas where clients support 5GHz and 2.4GHz is congested), and is far from a shabby device with the latest firmware installed (available free of charge on the HP site the last time I looked - HP took over 3Com). However, organisations with old 3Com controllers who are trying to keep their systems going a bit longer might drive up the prices of surplus 8760s.

There's currently an 8760 available Buy It Now for £39.95 with no power supply. The power supply is no great loss: the 3Com power supply was an awful device that wasn't properly compliant with the 802.3af Power over Ethernet standard (putting voltage on the line without following the 802.3af algorithm of ensuring a PoE compliant device was connected was a bad idea!). Any 802.3af or 802.3at power injector will do to power these units - expect to pay around £20 new for such a device.


If either of the RADIUS based approaches seems worthwhile, this page (which I haven't reviewed for accuracy, but which directly addresses the configuration issues required) may well be a good place to start reading.


Recognising the limited budget (if any) for this task, and the need to limit technical complexity, I suggested an open source system that can be experimented with for free and deployed on relatively modest hardware in place of the existing router. An off the top of my head guess for deployment costs on new hardware would be £300-350 - but who said new hardware was required? Once you know what you're looking for, you may find a suitable computer and switch on eBay, as people discard older kit to move up to something newer and more powerful. There are also certain commercial firewalls that companies are discarding that can be used to run pfSense can be installed.

To start with, an old PC with a couple of NICs will do fine, though power consumption, noise, space and reliability might mean an old PC is not the best long-term solution to deploy pfSense.

I think, (dangerous I know), that we can keep hypothesising til the cows come home or, we get answer to ian72's question in Post#2.

I know it sounds obvious but honestly the easiest way forward is to get an unlimited ISP which is just far far easier in house shares. With game downloads via stream, Netflix etc you can rack up 100s of GB fairly easily.

In reply to a post by ukhardy07:

I know it sounds obvious but honestly the easiest way forward is to get an unlimited ISP which is just far far easier in house shares. With game downloads via stream, Netflix etc you can rack up 100s of GB fairly easily.

Read the questions asked in Post#2. It may be a wireless connection, not in this country.

In reply to a post by Stevenage_Neil:

I think, (dangerous I know), that we can keep hypothesising til the cows come home or, we get answer to ian72's question in Post#2.

True - though if the provider is a mobile one, pfSense works 'out of the box' with many USB mobile devices.

At a guess, the provider is either this Indian one (could be mobile or fixed) or this one in Jersey (likely to be mobile).

A lot happened since I posed that question. Have to say I liked your reponses but I have been in IT long enough to know what you are talking about even if I haven't personally done these myself.

But, still waiting to hear from OP as to what the position with the ISP is as that is fairly critical (and I suspect will rule out unlimited).

Might have to have a look at pfsense myself in a bit of spare time just to have a play - don't actually need the functionality but sounds interesting.

In reply to a post by ian72:

Might have to have a look at pfsense myself in a bit of spare time just to have a play - don't actually need the functionality but sounds interesting.

I rate pfSense because I find it does the same job as expensive proprietary solutions using relatively modest hardware, with the bonus that the software is free.


Choosing pfSense was natural for me because of my familiarity with the underlying FreeBSD operating system. If I find a problem with pfSense, I can often fix it myself and contribute the patch back to the pfSense project. That's one advantage of open source - the source code is there and you are free to make your own changes.

If DIY support is not an option, there is an extensive support infrastructure, and commercial support is inexpensive. If you land up paying for a small amount of consultancy, you're still likely to save a lot compared to opting for a commercial solution. If you allow any patches to be contributed back to the project, everyone benefits.

The bug find rate is a higher than usual at the moment, because the recently released pfSense 2.1 adds IPv6 support and has moved to a more modern underlying version of FreeBSD. pfSense 2.0.4 is still available if you want a mature IPv4 only version.


I'd like to see pfSense spread into the consumer marketplace, but it is currently limited to x86 and x64 hardware by the still immature ARM support in FreeBSD. There are some neat little x86 single board computers that can run pfSense from the likes of Soekris and PC Engines, but current generations of these boards struggle with the faster connections people increasingly have, also the limited RAM on these boards precludes the use of certain higher footprint packages (especially Snort and Squid) in all the but their most basic configurations. It was with these limitations in mind that I suggested a fanless Intel Atom board with an Intel Gigabit NIC as an excellent platform for pfSense from a price / performance / running costs perspective.

As FreeBSD on ARM matures, it may be possible to run pfSense on some consumer router hardware in place of the manufacturer's proprietary solutions, but this is some way in the future at best. pfSense on Raspberry Pi is a nice idea, but is likely to be unfeasible because the Raspberry Pi uses a relatively slow USB connected Ethernet controller.

Ahhh I see, thank God the UK has a tonne of unlimited options.

I really would like to know who you think your target audience is.

The post with the options was at an unavoidably high technical level for reasons explained in that post. Either I posted at that technical level, or I could offer no solutions.

The follow-up post about pfSense was in reply to someone who identified himself as comfortable with that technical level, and who expressed an interest.


I've heard your complaint that the technical level of my contributions to this thread is too high because you don't understand them despite the obvious technical ability as indicated by your degree. As you acknowledged yourself, things have progressed fast in networking - a lot has changed since the original development of Ethernet, though that work remains important and seminal today. Just Googling many of the abbreviations will illustrate my reply.

I notice you haven't offered a solution of your own to the problem posed, let alone one at a lower technical level. I've tried to encourage anyone who wishes to experiment with my suggestions by lowering the barrier to entry (suggesting the use of a free GUI driven system rather than a 'bare bones' operating system, offering some pointers to relevant information and making a suggestion for inexpensive equipment from eBay that I know from personal experience will work). Unfortunately, unless I've missed something, we're in an area that's either DIY with self-education or involves a high price tag.


These forums contain everyone from those who class themselves as IT novices through to engineers with far greater expertise than me. We are in a technical discussion sub-forum here, and I don't see anything wrong with using the lowest level of technical content appropriate to the question asked.

If I see a post that I don't understand on these (or any other) forums, I ignore it, carry out my own research in an attempt to understand it, or ask a relevant follow-up question.

You can ignore my posts silently without adding to the noise in this thread, as I wasn't replying to you. As you will see elsewhere on the forums, I try to keep the technical level in my posts down wherever possible.

In reply to a post by David_W:

I notice you haven't offered a solution of your own to the problem posed......

Er.....

In reply to a post by Stevenage_Neil:

I think, (dangerous I know), that we can keep hypothesising til the cows come home or, we get answer to ian72's question in Post#2.

Note the word hypothesising.

Here's a simple solution: Get all residents to use Windows and install Networx on all machines.

Edited by deleted (Mon 07-Oct-13 15:04:58)

In reply to a post by Stevenage_Neil:

In reply to a post by Stevenage_Neil:

I think, (dangerous I know), that we can keep hypothesising til the cows come home or, we get answer to ian72's question in Post#2.

Note the word hypothesising.

I did. Even if the OP is on a mobile network, the answer I gave remains valid.

If going down any pfSense route, most USB mobile broadband devices just work - plug the device into a USB port on the pfSense box and fill in the details on the cellular and interface pages of the pfSense interface. If that approach doesn't work, either move the SIM to a pfSense compatible device (if using a SIM based service), or interpose pfSense in a bridged configuration between the mobile broadband device and the client network (use an Ethernet cable to the existing router if you can, otherwise a wireless bridge if the mobile device is purely wireless).

If going down the wireless LAN with RADIUS accounting route, the best option is to configure RADIUS accounting on the existing wireless LAN if you can, though it's unlikely a consumer grade device has the necessary support. If you need to use a separate access point for the clients that has RADIUS accounting support, connect that access point to the mobile device using an Ethernet cable (preferred solution), a dual band enterprise access point in a client / access point configuration (in some cases you can use a dual band access point in a 2.4GHz client and 5GHz access point configuration - or vice versa if the mobile device has 5GHz support), or use a wireless bridge against the mobile device's wireless.


Again, this is only a high level answer. Moreover, these details don't matter much. My original answer gave some broad approaches that were equally applicable to a fixed, wireless or mobile service. If the original poster gives more information, it will be possible to eliminate many of these options and give a more precise answer.

For what it's worth, I join those who believe the best answer is an unlimited tariff, rather than any of these technically complex solutions. However, as we appear to be talking about a foreign country, such a tariff may not be available at an affordable price.

In reply to a post by Stevenage_Neil:

Here's a simple solution: Get all residents to use Windows and install Networx on all machines.

Will that monitor the phones and tab?

Amazed you lot are still discussing this after no responses from original poster.

In reply to a post by BatBoy:

In reply to a post by Stevenage_Neil:

Here's a simple solution: Get all residents to use Windows and install Networx on all machines.

Will that monitor the phones and tab?

It won't. This approach is my second bullet point back in my earlier reply, where I discussed other drawbacks.


stevenage_neil's point that I'm ultimately responding to here, with the addition of BatBoy's commentary, was edited into the post after I read and responded to it.

As MrSaffron says, there's nothing more we can do without the original poster. This thread is rapidly heading for my wombat filter if he doesn't come back. (wombat (slang) - waste of money, brains and time).

In reply to a post by MrSaffron:

Amazed you lot are still discussing this after no responses from original poster.

Didn't I say that earlier?

In reply to a post by Stevenage_Neil:

I think, (dangerous I know), that we can keep hypothesising til the cows come home or, we get answer to ian72's question in Post#2.

Maybe other people are interested in the answers? After all, if the discussions are only for the OP it would be better to use PM's like the mods do on the BT Community forums.

In reply to a post by BatBoy:

In reply to a post by Stevenage_Neil:

Here's a simple solution: Get all residents to use Windows and install Networx on all machines.

Will that monitor the phones and tab?

Only Windows based machines as far as I know.

Edited by deleted (Mon 07-Oct-13 18:15:29)

Well, that's not going to meet the requirements of the OP whereas the posts you didn't comprehend will.

In reply to a post by BatBoy:

Well, that's not going to meet the requirements of the OP whereas the posts you didn't comprehend will.

Read the post again, the operative word being Windows.

In reply to a post by Stevenage_Neil:

Here's a simple solution: Get all residents to use Windows and install Networx on all machines.

I know of no software that does this but implementation wise, the only way I can realistically see of doing it would be to have a packet sniffer/analyser running on the LAN interface of the broadband router (or if not that, a dedicated machine connected to a mirror port on a switch so that all network traffic on the LAN port the router is connected to can be sniffed). It will be possible to record every single Ethernet frame that it has sent to clients and received from them. That being the case, it will then be possible to graph per client MAC address the inbound and outbound data sent and received per unit time per client.

As I said, I know of no software which does this (maybe someone else does). But I would imagine such a solution will involve PCAP sniffing the router LAN interface at raw layer 2 so as to record all client MAC addresses seen and their corresponding usage (from the size of their Ethernet frames and quantity).

The annoying thing is even MAC addresses can be falsified so if people really wanted to be awkward, they could start randomising their MAC address rendering the layer 2 analysis completely useless (how would you then know what devices these MAC addresses represent?)

It is a tricky problem to have a guaranteed solution to (in my opinion).

Edited by deleted (Mon 07-Oct-13 18:54:09)

In reply to a post by Stevenage_Neil:

In reply to a post by BatBoy:

Well, that's not going to meet the requirements of the OP whereas the posts you didn't comprehend will.

Read the post again, the operative word being Windows.

In reply to a post by Stevenage_Neil:

Here's a simple solution: Get all residents to use Windows and install Networx on all machines.

Maybe you should read the requirements specified in the OP?

In reply to a post by mixt:

I know of no software that does this but implementation wise, the only way I can realistically see of doing it would be to have a packet sniffer/analyser running on the LAN interface of the broadband router (or if not that, a dedicated machine connected to a mirror port on a switch so that all network traffic on the LAN port the router is connected to can be sniffed).

You're describing one implementation of my third bullet point above. Arguably it's better to direct derive statistics from the router (i.e. NetFlow / sFlow or similar) than tee off all the traffic for external analysis (the mirroring approach you describe), but assuming the switch you're using for the mirroring is competently mirroring all the traffic without dropping anything, the outcome is the same.

In reply to a post by mixt:

The annoying thing is even MAC addresses can be falsified so if people really wanted to be awkward, they could start randomising their MAC address rendering the layer 2 analysis completely useless (how would you then know what devices these MAC addresses represent?)

That's a good argument to go for one of the RADIUS based approaches I gave. In those approaches, the devices must authenticate to the network before being authorised to gain access, and the accounting data will be linked to the authorisation.

It's near certain that you would want to use the same RADIUS server for authentication, authorisation and accounting (AAA), though there are certain scenarios where it makes more sense to use a RADIUS server for accounting only.


If you can be bothered to implement it correctly, you could go as far as deploying a wireless setup WPA2-Enterprise with EAP-TLS. This requires each device to have a client certificate issued by a (typically local) certificate authority. The management overhead of EAP-TLS means other password-based forms of EAP are more common - EAP-TTLS if you have device support, PEAP if you don't. You can use different forms of EAP on different clients if you configure the RADIUS server correctly.

The Enterprise versions of WPA and WPA2 are crytographically harder than the Personal (PSK) versions, as each client has its own unicast keys, and both the unicast and group keys are rotated frequently. If possible, you should use WPA2 with both TKIP and mixed mode (which allows clients to use WPA rather than WPA2) disabled.

In reply to a post by mixt:

It is a tricky problem to have a guaranteed solution to (in my opinion).

Agreed - in an assumed hostile environment (I used to deal with a lot of secondary school networks - with definite hostile intent from a small sub-group of users!), you have to assume that only approaches that force compliance will work.