Debugging Our Network (Spoiler: Not Comcast’s Fault)

TL;DR

Skip to the end if you just want the tool to do this yourself, otherwise, enjoy the story about how we debugged a nagging network issue.

Corruption!

So, we’ve been experiencing some annoying network issues lately and we were almost positive it was our ISP (Comcast).   The issue would occur on large files (think major OS updates, video on demand, music downloads, etc).  Basically we would perform our 20 minute download and get stuck with a corrupted file.  Worse, we would pay for a VOD movie and start to watch it only to find it is a glitchy mess with intermittent sound.

Yikes!  Something had to be done.

How hard can it be?

But first — a little about our network configuration.   We have a Comcast cable modem that goes to our main firewall router that then gets distributed throughout our house via switches and wireless access points.   The initial hint was that we saw this corruption on both the wired (Tivo) and wireless downloads.

The worst part of all this is that the problem was intermittent.  Sometimes large files would work and other times they would not.  I needed data if I was going to convince Comcast that this was their problem (note the negative bias).

In the beginning

I started by manually downloaded the large file.  In this case, I leaned heavily on my buddies in Cupertino and one of their 1.1GB major software updates.  Thankfully, their update site provides a SHA1 checksum.    The process was this:

  1. Note the start time
  2. Download said file
  3. Note the end time
  4. Open a terminal window and type “shasum /path/to/filename.dmg”
  5. Make a note in some file whether or not the download was corrupt.

Did I mention

Oh yeah, each one of these downloads took me 6-7 minutes and I have kids now.   This manual data gathering was becoming a pain.   I needed relief and automation so that I could run these tests unattended all night long and gather the precious data in all the configurations and locations of our network.

Thank you Guido

Enter python and 30 minutes of time.  My son, Leo, thankfully cooperated by not injuring himself during those precious 30 minutes.   My daughter, Dagny, also helped out by taking a nice long nap.  The stars were aligned and all was go.

The script

So, I knocked out a quick and dirty little python script that does the above steps while also outputs the notes and results into a csv file.  Don’t worry, the downloaded GB file is deleted every time.  A link to the script is below (NOTE: You will need to tweak the download URL and checksum to suit your needs and desires)

Download_Large_File_examp.py

The results

So, after all the work, what does the CSV file look like when imported into Excel?

Output from the python downloader script with some conditional formatting applied.

Output from the python downloader script with some conditional formatting applied.

And the Winner is………

Surpise!  It was not Comcast!  In the end, I was able to trace the corruption to our Netgear FVS318G router which was running some old firmware (3.0.5 or something).   After installing the latest firmware (3.1.1+) the problem vanished.

The lesson here — it’s not always Comcast’s fault and it definitely helps to throw a little python around now and then.

SHeBeNoBIF

Definition

Strongly Held Beliefs No Basis In Fact(s).

Pronounced “She’-Ben-O-Biff”. Similar to NIH (Not Invented Here). SHeBeNoBIF can be a reason for rejecting something or ignoring new data, purposely remaining stubborn.

Examples

Our internal IT department has SHeBeNoBIF regarding migrating to external cloud services. The repbulican party’s SHeBeNoBIF keeps dragging their approval ratings lower and lower.

CREDIT:

Dr. John G Mooney    Pepperdine University     ca. 2012

Out in about 6,000

So, it’s been a long while, but worth an update (since in the last post we were at 5,000 and wondering when we’d be done).
Sometime in the last 4 months, our little daughter figured it all out, and before her 3rd birthday we were consistently diaper free (except for night time, of course) and completely ecstatic, too!

Of course, since that time little Leo was born and we are once again back into diaper-land with little to no sleep…….  So the next time I wake up, I might post something more intelligible.

Great Balls of Tea — Details of Spherification

Introduction

A recent order to the Spice House (www.thespicehouse.com) had me excited.  Not only did they have some smoked paprika and powdered sumac (good on top of humus), they also had odds and ends for molecular gastronomy.   In case you don’t know what that means, it’s the hoity-toity way of saying doing a little chemistry with your food to create unusual effects.  Candy making is molecular gastronomy based around sugar and it’s behavior at certain temperatures.

One of the most notable effects from this “rockstar” chef movement is Spherification.   I’ll warn you that I’m no expert and definitely not a historian on this manner.  In fact, apart from seeing it done on Iron Chef now and then, I’ve only been served spheres in a restaurant one time and it was not even that memorable.

But I wanted to give it a try — so I bought some stuff for making balls (er….spheres).

Spice House offers excellent small-sized bottles for spherification.

As I mentioned earlier, Spice House offers small-sized amounts of Calcium Salt and Sodium Alginate.  Quantities sized right to get started, and the ingredients are food grade (so not from some scary chemistry shop where they may be tainted with ferric chloride, cyanides and other nasty not-so-edible chemicals).

Why write a post about my balls?

My stuff showed up just in time for some weekend play — but now what?   I trolled the web, but couldn’t find any straightforward explanations or recipes.  How much do I mix in?  How do I make the drops?  What are some basic tell tale signs.   Give me some range of expectations a la Good Eats so that I can tune in my spheres and get this process working.

I gave up on the internets and started playing and making my own notes.  What follows are my experiences from a weekend day of playing around with little alginate balls.

The Process

I didn’t make that much of a matrix, as I was basically trying to dial it in (and my 9 month old can only handle being ignored for so long).   The great part, though, is that I was able to use things that are readily available to any home cook.  Oh, and I took lots of notes of both the successes and the failures.

The Solutions

To make spheres, you drop one solution into the other.  Typically, the flavored ingredient is mixed with Sodium Alginate and dropped into a solution of Calcium Salt.  Through some chemical magic, drops turn into little spheres with a harder outer coating and a soft liquid gel like inside.  Properly done, the little spheres pop in your mouth like caviar and release tastiness on your palette.

Ca:Salt Solution

The Calcium Salt solution is easy to make up.  In my case, I just used room temperature tap water (our tap water is tasty good) and whisked in a small amount of Ca to dissolve.

Sodium Alginate Solution

The sodium alginate solution is a little more tricky.   For this set of experiments, I made spheres using sweet tea (thanks, Steph).  I figured it was a good basic starting point, as it is failrly neutral in acidity and we plenty of it to work with.

The problem with the alginate lies in actually mixing it together (emulsifying).   I first tried room temperature sweet tea and a whisk.   This just made clumps, which don’t work and are not good eats.

The answer: I popped the clumpy mixture into the microwave for 30 seconds and took it just shy of a boil — voila, the sodium alginate clumps broke apart and a whisk finished the job.    The resulitng liquid was fairly bubble free and slightly thicker (think maple syrup).

Another method of emulsification that I have read about and tried is to use a hand stick blender.  This is an aggressive approach and definintely worked at emulsifying the liquid.  However, it also seemed to incorporate lots of bubles and it loosened up the liquid considerably.  Using the hand blender, I needed more alginate per liquid volume in order to achieve the desired thickness.  Also count on a period of rest (or hook up your shop vac and a home made bell jar) to de-air your mixture.

The Dropping methods

Everywhere I looked online people used various tools to create the drops that create the little spheres.  You can use a toothpick or a spoon or a syringe.   As I worked with the various methods, I quickly developed opinions — which I will share with you below:

Toothpick

Good for initial testing of your solutions to see if they will sphereize.   Terrible for creating large amounts of balls — 1 ball every 5 seconds is pretty maddening.   Also ends up creating somewhat inconsistent ball sizes, depending on how much the solution clings to the toothpick.

Spoon

Very uncontrolled.  Pretty much useless for creating spheres.   You can create noodles and spermy looking shapes, however.

Here are the 3 dropping tools I attempted to use. A small eye-dropper type bottle on the left, a more expensive version on the right and a syringe front and center.

Syringe

Maybe I had a low quality syringe, or too small a syringe.  I had a devil of a time creating consistent drop after drop.  Instead, the syringe would clog now and then and I would end up spraying out some alginate solution into a rats-nest in the calcium salt.   Taking in a little air into the syringe helped a little bit with constant pressure against the alginate solution.  Either way, I was not fond of the syringe method.    The best use of the syringe was to cleanly fill up the dropper bottles.

Dropper Bottle

After getting frustrated with the syringe, I tried 2 different dropper bottles.  Both of these worked many times better than the syringe.  They created consistent sized drops and did so very very quickly.  The second dropper bottle was able to create 100’s of droplets in a minute!   A good dropper bottle is easy to fill and the nozzle won’t clog with alginate.   Too tight a nozzle and and the alginate eventually gums up the orifice and you can’t make spheres anymore.      In the end, I used a syringe to cleanly fill a dropper bottle and unleash a batch of droplets into the calcium salt solution.

Example dropper bottles that I tried:

  • (dropper bottle in picture on right) VWR International:  16354-400   $80 for qty 12
  • Low density VWR International 46300-592 or larger (46300-594 is 8oz),  $33 for qty 25

The other nice thing about the dropper bottles, is you can cap them and store your alginate solution for up-to-the-minute use.   One important note about spheres is to not keep them sitting around too long. After about 30 mintues or so they eventually go “stale” and harden up into solid balls — not nearly as texturally interesting as caviar-like spheres.

More Detailed Notes And Quantities

Alginate solution on the left and calcium salt solution on the right -- we're ready to make some balls.

Below are my notes on the solutions and mixes and results.  After trying to weigh out the ingredients on a precision scale, I gave up — how many of us actually have scales in our kitchen accurate enough to measure fractions of an ounce (or just a couple of grams).  In this case, volumetric measurement is going to be more reliable and more available to the standard home cook.

Sodium Alginate Solutions

Sodium Alginate was added to the room temperature sweet tea, but since it faield to dissolve, the mixture was heated in the microwave for 30s – 1 min (just shy of boiling) and whisked to emulsify.

Solution                  Sweet Tea                   Sodium Alginate

A                                   1 oz                                1/8 tsp

B                                   2 oz                                1/8 tsp

C                                   2 oz                                3/16 tsp

D                                  3 oz                                 1/4 tsp

Calcium Salt Solutions

Calcium Salt was added to cool tap water and dissolved with a whisk.

Solution                       Water                            Calcium Salt

1                                  8 oz                                 1/2 tsp

2                                  4 oz                                  1/2 tsp

3                                  8 oz                                  1/4 tsp

Results and Notes

In the following section, we’ll refer to the solutions above.   Obviously, “A1” means sodium alginate solution “A” dropped into calcium salt solution “1”

Red food coloring and simple syrup make these early spheres visually pop off the plate.

A1 Observations

  • Time in Solutions:
    • Drops 1 minute in solution were a little soft, but hardened up a little after sitting.
    • 3 minutes in solution was perfect
    • 4 minutes in solution was a little too hard
  • Solution Notes:
    • Use a little less alginate, the “A” solution was noticeably thicker — almost mayonaisse.
    • Probably could use some more calcium salt since it took so long in solution
    • There was not an excessive salt flavor from the calcium.
  • Syringe Notes:
    • I used the syringe + louver, and got about 3mm drops, but the syringe kept clogging
    • Once I got some air in the syringe behind the alginate solution and it helped with consistency.
  • After 30 minutes of standing by, the spheres were still edible

B2 Observations

  • Solutions Notes:
    • The Calcium saltwater was quite strong
    • Not enough alginate to form a ball
    • Very strong resigual Ca:Salt on spheres….must be rinsed well
  • Drop does not really penetrate surface of the water to make a sphere…instead it sits on surface and additional drops glom on.

A mound of perfectly popping prismatic sweet tea spheres sits on a white plate. Super tasty and sweet!

C2 Observations

  • This combination worked pretty well.
  • Some balls were solid however, way more than A1 — this is not as appetizing
  • Solution Notes
    • Required less soak time than A1
    • The #2 salt solution definitely needs a good rinse.
  • Dropper Notes:
    • Used the dropper bottle (“methonal bottle”)   VWR International:  16354-400
    • This was great for a consistent drop size, although if the alginate solution gets too thick it can sometimes clog.
    • Used the syringe to cleanly fill the dropper bottle.   This worked really well.

D3 Observations

  • Time in Solutions:
    • After 1 minute in the Ca:Salt solution, the spheres were nice and delicate with a good outer layer and  a snap when you bite into them.  Perfect mix for our sweet tea spheres.
  • Solution Notes:
    • Alginate needed to heat or brought to boil for faster/easier emulsification
  • The drops enter the water almost toroid shaped but did sphereize.
  • Dropper Notes:
    • Tried a different dropper bottle (like a visene eye-dropper). Although a little more difficult to fill, it did NOT clog.  It also generated dozens of spheres quickly — just squeeze.
  • Stayed good for 5 minutes (easily) after pulled out and rinsed and dried.

Other Notes and Findings

Drying

Drying these little buggers is in itself an art.   I tried a couple of ways, but by far the easiest was to spread them out on a paper towel in a single layer and cover with a paper towel and slowly roll the balls between the two layers.   The other method is to creeate a “sling” out of a few paper towels and kind of toss the balls around inside the sling.

Spheres, balls, everywhere. Watch out -- these guys know how to roll!

Balls Everywhere

Expect to find and lose balls everywhere.  Once loose, they roll….and because of their size and transparent quality, they can be extremely difficult to find.

Serving

Serve as a presenation note on the side.   I’ve served in leiue of sauce on top of fish (think parsley puree spheres with a good touch of salt as a high note on top of salmon).    Concentrated flavors and colors work well.

Tastes for Serving

For saucing or as a side note, think big bold flavors.  The spheres are small and you want them to pop and release bundles of joy.   I made a nice parsley sauce which, when over-salted, produced a wonderful topping for fist.   Blueberry juice is a little weak, but blueberry preserves thinned with water carries a good punch of flavor.

Other Hints

Mix up your alginate and drop all of your balls at once into the Ca:Salt.  Insead of trying to fish spheres out of the Ca:Salt, just pour the entire Ca:Salt solution out and into a strainer.

3oz of alginate solution is a LOT of spheres.   Easily enough as a side note on plates for 4 people.

I tried using a scale to measure ingredients, but this proved frustrating, as most scales simply cannot handle small fractions of an ounce (or gram) quantitiies accurately.

Debugging

If the drops sit on top and do not sphereize, then there is a likelihood that you don’ thave enough soldium alginate.   If, after a minute, the balls are still too fragile, up the calcium salt solution.

Next Steps

So what next?

Well, I’ve made some great spheres using a celery salty sauce and put this on broiled salmon – but I used the hand stick blender to emulsify the alginate and it took far more alginate than expected.   So clearly, there are some variables at play.   Perhaps I need to buy a pH meter and do some work with acidity and quantity other variables.

The metric that seems key, but is the hardest to quantitatively measure, is the viscosity of the alginate solution.  This is definitely a case where having some experience regarding what to expect and mixing it by eye may be easier.

Welcome to the NEW blog.redstoyland.com

Well…. a weekend of work and I hope that the migration from static publishing software to blogging software will enable me to make more faster bettah quicker posts.  We’ll see.

Although I have tried to maintain links, I expect that some of the older posts might have broken links.   Worse, the main Reds Toyland site is definitely going to have some orphaned links that no longer can find the old reference.  Oh well — not that big a deal.

Favorites (at least what people tell me they like reading) are geting their own category.   These include the awesome My First Hospital Visit post and the Open Letter to the Diaper Changing Mom rant.

Enjoy the new site!

Goalie Thoughts: I Has A Flavor

Today, we’ll discuss one of those less-than-obvious things about being a goalie (and player, too perhaps). Red is Smoking Hot with steam coming off his back.

::Background::

While I have short hair, I’m currently sporting the (out-of-style) spiky look. To get said spiky look one has to use product. Not a lot, mind you, but some reasonably stiff hair gel of sorts. Continue reading Goalie Thoughts: I Has A Flavor