There is a very peculiar character in Sesame Street called “The Count”, he is a Vampire with an Eastern European accent that always introduces himself like this: “Greetings I am the Count… do you know why they call me the Count? Because I love to count things”.
Well, I love counting things too. I have implemented a counting system with my daughters based on a whiteboard. Every time they finish a chore or a task they get one point. They love it because that usually translates to some amount of money they can spend in whatever they want.
But I don’t want to be the police of points. I rather automate that. Also, I’d love to provide them with some sort of feedback in a leaderboard for them to know their progress.
There are 2 ways I want to count things: a. By them reporting them via a voice recognition interface, b. By them be auto detected when they happen.
The count via voice has an extra requirement. I don’t want my daughter’s voices to go out to the internet so the whole solution needs to run offline.
Implementation Details for the Voice Recognition counter.
Somebody in the house presses button connected to B. When that happens, A starts listening for 5 seconds. Just enough to grab the voice blip : “I washed the Dishes” that is placed in variable X . X is passed to a voice to text converter (1) running in a Raspberry Pi that puts the result in Y. The RPi also runs a voice classifier to determine who’s voice it is, puts the results in W. Then a text classifier determines what chore has been reported and puts it in Z. The author of the chore ( W) and the chore type (Z) are used to increase the total count for that person on that particular task (T). At the end the Total, the author and the chore are put together to be shown in the display M.
Implementation Details for the Automatic Counter
This is a much simpler version of the Voice Recognition counter. When C passes by D there is a match. There is a small tag in the dog’s collar that is detected by D. By doing so, D already knows who passed by. This is used to determine the total count (T). Then it is displayed in M.
The second option is to do the opposite, let’s assume that the bike has a small Near Field tag (E) that F is aware of. When E no longer detects E, that a means that I took my bike for a ride. That increases the count (T). That it is displayed in M.
Some design considerations:
⁃ The voice recognition requires a separate process to train the model to be able to classify voices.
⁃ You only need one display M, it could just rotate the count it is displaying every 10 seconds.
⁃ The counts need to be reset periodically to avoid error drift. Once a month seems to be a good compromise. The historic can still stay in memory and later we can figure out a better way to display it.
⁃ If there is a new thing to be counted, you just need to start sending it to the increaseCount function. If it is new, it will just create a new register for it.
⁃ The whole system should be intentionally designed to be off-the grid. No internet connection or wi-fi communication should be needed.
So, what do you want to count now? What you think about Push Up/ Pull ups, burpee repetitions? That wouldn’t be that complicated to implement using a cheap microcontroller with an accelerometer tied to an armband. I still can’t believe my Apple Watch can’t do that…
Dr. Brown from Back to the Future was one of the reasons I decided to be an engineer. Not because he invented a time machine (pfff) but because of the machine he created to feed his dog (Einstein). If you haven’t seen that movie, stop reading this and go watch it.
Funny thing is that while some of my other friends in Engineering school wanted to become engineers because they wanted to go to the moon or something as far fetched as building a sentient cyborg, I just wanted to build cool gadgets for myself.
In this Blog Post I present a quick draft of how could we build a dog feeder using technology I’m familiar with: Microcontrollers, sensors, a mesh network and off the shelf mechanical components.
The basic idea is to have a mechanical timer/relay like those used in gardens advertise if it is time to do something or not. The way I would implement this is by reading whether current is passing or not (sensor N). Yes, I would have to translate from 220V to 5V but that’s not a big deal. I could just use a 220v to 5V transformer ( also known as a phone charger) to generate an input signal.
After that, I would check whether there is food in the main container. I was thinking about a sonar (sensor B) , I’ve worked with sonar sensors in the past and they are pretty reliable once you calibrate them. Here, it would know if there’s food based on how long it takes the ping to come back. A value of 2 is equivalent to 20% of the container being empty. Sonars don’t rely on light so I assume it will work with some food dust around.
In addition to checking if there’s food I need to verify whether there is food in the plate or not (sensor I) . I would place a small light sensor on the bottom of the plate. If light is detected, that means there is no food on the plate.
If all looks good I would start the motor that drives a dispenser screw (actuator C) for it to drop some food to the pipes that are already pressurized by the air pump (actuator D). Similar to when you vacuum your own home, it will drive the food kernels through the pressurized pipes all the way to the element F.
A vibration sensor (sensor E) detects whether food is hitting the walls of the pipes. If no vibration is detected that means no food is in the pipes and something must be wrong: Either no pressure or no food. We output an error message to the message board (actuator M) and exit the program. Sorry dogs, no robot serviced food for you today.
Once food makes it to the intersection F, it will go either left or right. Actuator F is just a ramp that moves to one side or the other letting the food fall in Chi Chi’s plate or in Hugo’s plate.
Maybe this is an overkill, but there’s yet another sensor in the pipe that detects whether food is passing through all the way to the plate. This is to confirm that F did its job properly. I could just instead check sensor I to figure out if the plate is filled in.
Because dogs also need water to live, specially after eating that processed food, we check the level of the water using a floating magnet. A magnetic field magnet checks the intensity of the field, infers how close the magnet is (sensor K) and determines the level of the water. If water is needed, the water valve Q is opened for a certain amount of seconds to fill the bottle.
And all this so we can focus on the Flux Capacitor…
A very curious device that I found in every apartment when I originally moved to Brooklyn, was this white circle shaped plastic protuberance hanging from the ceiling in almost every room. I must mention that back then, I also moved in with my own black circle shaped metal device also known as a “comal”. A comal is a cast iron flat plate to cook everything from tomatillos, cecina to tortillas and everything in between. It was not long until I discovered that both of them didn’t like each other. Every time I turned the comal on to prepare a midnight quesadilla, the white protuberance on the ceiling started howling as if I was a thief. What was so important to guard that couldn’t wait for my midnight snack to be ready? Now I know, it was my own life.
I was about 8 years old on a christmas family reunion when I discovered the smell of kitchen gas (I figured later, that smell was added on purpose). I remember standing there in the kitchen inhaling what I assumed was a delicious toast bread smell. My aunts rushed in and made a big deal because gas was leaking, opened the windows etc. Additionally they decided, that it was a good moment to give little R a good life lesson and proceeded to give me a bunch of facts on how dangerous gas was, how many people die every year from poisoning, etc. From that moment on, I became obsessive about it, always (literally every day) checking whether the stove was not on before going to bed or when coming home. The irony is that all this time I’ve missed the main cause of death for this kind of accidents: Carbon Monoxide. Assuming no explosion, Natural gas kills you the same reason water kills you when you drown: Lack of Oxigen to breath. You’ll get drowsy before it is lethal. On the other hand, Carbon Monoxide kills you because it attaches to your red blood cells resulting in your body not being able to capture the right amount of oxigen for a prolonged amount of time (1). In the former, the risk stops as soon as you leave the room (on time), the latter: it’s a death sentence if you are not treated fast enough.(3)
If you search for CO sensors on the internet you’ll realize they are extremely affordable. So, if CO is easy/cheap to detect, why are there still horrible accidents happening?
Burning Natural Gas (properly) does not produce Carbon Monoxide. That means you can cook a soup for 4 hours straight in the middle of the winter with the windows closed at no risk. However, a bad flame (usually other color than blue) or preparing quesadillas in a comal, using an iron cast skillet or a poorly maintained gas water heater indeed produce CO (1).
What to do about this? First of all, get a couple of those white rounded plastic protuberances (CO sensors) and install them in the kitchen and rooms. However, detecting CO is half of the solution. A smart home should be able to do something about it in addition to sounding the alarm. My proposal is to remove the human from the loop. Same as AirBags in a car that just act when there is danger to protect the passenger, something in the kitchen or room should just act upon the threat of excessive Carbon Monoxide.
Following: a far fetched concept on how to solve this problem, we need to start somewhere right?
When you’re in a gas station, pay attention to what’s on the ceiling. A massive installation of something that looks like Medusa’s hair, it is a series of hoses pointing to you that will throw fire extinguishing foam to you and your car in case of fire. The point I’m trying to make is that in emergencies, everything is valid: Even soaking your client with foam (sounds like a Punk’d episode). What I’m proposing as a concept is something a little bit radical: To suck all the air in the room instantaneously and expel it out of the kitchen replacing it with fresh air from outside. It doesn’t even matter if outside is 0 Celsius (32F), what matters is that the room got detoxed and you’ll be fine.
Plan of Action
In a nutshell what I’m proposing is to have a return air fan and a set of separate dampers that will open at the dangerous CO levels event to let toxic air out and fresh air in. The fan needs to be big or powerful enough to recycle the room’s air fast enough. A better option is to be a little less radical and have the system detect CO early on and start moving air out before it becomes dangerous. In either case I need to make a hole on the wall or on a window for the air to flow in/out without having to depend on a sack of flesh (the human) to open the windows.
Regarding the actual CO sensors, I’m not a big fan of reinventing the wheel (although that’s what we do in this blog all the time), for that reason, it would be ideal to reuse the already commercially available ones. Maybe we could have a microphone that detects the loud alarm sound, to trigger the air flush sequence. As usual, humans (like me), will press the “silence” button in the alarm and keep preparing the quesadillas however, by then the CO Flusher would have already started the sequence of opening the dampers and start the exhaust fan. That way we get the best of both worlds: The human has been notified, and there is an actual automated action going on to clean the air.
In commercial buildings you have two independent air ducts: One for AIR IN and another one for AIR OUT. Additionally, there are cooling and heating coils in the air return to remove humidity from outside air and to heat/cool the air before it is returned to the room. However in this case, we use the same duct for IN and OUT intermittently to avoid the hassle of installing two ducts.
The most basic setup consists of a fan microphone and orchestrator, the dampers are optional but needed to keep the room insulated from external weather.
The internal CO sensor is used as a backup and to validate the alarm.
The orchestrator is needed to translate the sound into an event that will trigger the servo controller to open/close the dampers and for the drive controller to start the fan. Additionally, it approximates how much air flow has passed through the duct and reverses the motor for it to bring in fresh air.
+ A set of temperature sensors and cooling and heating coils for incoming air to be at room temperature
+ A set of humidity sensors for outside air to be dehumified
+ A smarter orchestrator that could turn this prototype into a mini HVAC system.
Top of Mind Conclusions
Installing a system like this into an existing house or apartment would be a little bit complicated, however in new constructions it could be part of the default installations making it transparent to the user. This system could also be useful in small houses that require indoor fire burning (stoves, fireplaces, etc) and in any kitchen in the world where quesadillas are prepared…
The idea in a nutshell: To install a series of levers, buttons, lights and sounds actuators around the house or apartment for dogs to be entertained while owners are not around. Ideally, the smart home would randomly generate new ways to call the dog’s attention and learn from what is effective or not.
Why would we want this? Let me use my own experience as an example : My dear dog “Chicharron” has destroyed about 15 books, 8 DVDs, a passport, a couple hats , a couch, countless toys and even a snake plant with his teeth. At first I thought he was just a bad dog, then I got the feeling that he was just hungry so I started feeding him much more. He still chewed on random things all the time (I get he is young and his teeth are growing). We gave him a massive amount of chewable toys, all colors and sized but nothing really changed. I’d like to believe he is just bored. Chewing things seem to be the way he copes with boredom because he never does it when we are around.
Doing some research I found a Swedish Dog games designer called Nina Ottosson, in her website she mentions the following:
“All dogs need to use their head sometimes in order to feel good, and some dogs have a greater need than others. If they don’t get to channel their energy into an organized activity, they can sometimes create their own “fun”, which is not always appreciated by the owners, such as chewing on things, or just get hyperactive”.
Worth mentioning: Nina Ottosson has a full line of Dog Activity Games that look great. I’m ordering a couple today for Chicharron. However what I’m proposing is a little bit more on the crazy Emmet “Doc” Brown side from “Back to the Future” with electronics, lights and mechanical components. If you remember the opening scene of the first movie, there is a full contraption that feeds “Einsteing” (his dog). I like to imagine that in this alternate and fictional movie universe, the “Doc” would have also invented and developed a way to stimulate “Einstein’s” brain (pun intended).
First of all we need to think about the Dog Interface/Experience (DI/DX):
The levers and buttons needs to be rugged enough to withstand chewing.
Dog should be able to interact with its Paws, Tongue, Teeth or Snout.
Light actuators should be within the dog’s line of sight.
Reward dispensers should make it easy for the dog to retrieve its payoff (and communicate explicitly that a reward was dispensed).
Sensors should not only include levers and buttons but also microphones (to pick up dog sounds like whining, crying or barking) and cameras for computer vision to be able to recognize bad habits (like taking things from the top of the table or opening the Trash can).
The feedback shouldn’t only come from the light actuators but also from any other smart home device (like living room lights, colors, blinds, etc)
Plan of Action
We are going to design an hypothetical system that will give cues to the dog(s) to take an action. The dogs will have to be trained to respond to the cues via the input gadgets (rope pulls, cranks, buttons) for them to acquire a reward (could be a treat or a sound). The system will get feedback (how and when the dog reacted to the cues) via the Microphone, Computer Vision enabled camera and the Movement sensors. A “Dog Actions Classifier” will interpret the feedback inputs and recommend the next cue.
The system should NOT be connected to the internet and that is OK. This is not intended for dog owners to play remotely with their dogs… This is literally, the Smart Home playing with the dog and learning from it.
Maybe it is not explicitly depicted in the diagram but all the components should communicate wirelessly. This would avoid the problem of dogs chewing on the cables.
The only components that need to be dog-ruggedized are the input gadgets : the cord pull, crank buttons, etc
Also, not explicitly depicted in the diagram but there is a micro controller that orchestrates all the gadget in an asynchronous way. Every gadget should still work even if that controller is down.
The most basic Classifier should be pre-trained, however it would be fascinating to have it train against every dog’s particularities. Also, to randomly come up with new cue-action-reward combos to discover new ways to entertain the dog.
The dog will reinforce the system the system will reinforce the dog.
+ Connect the system to the internet for owners to play remotely with its dog. Although I suspect owners will rapidly loose interest after doing so a couple of times. Hence the importance of having the smart home interact with the dog directly
+ Maybe implement something similar for cats? Do they care?
Top of mind conclusions
The chances of somebody investing more than a $100 in a toy or a system to entertain a dog are practically zero (I usually wouldn’t). Nevertheless, as I’m writing this post, Chicharron just destroyed today the living room rug that costs much more than a $100. Could the case be made that the ROI for such implementation is positive? How do you know what your dog is going to destroy in advance? Also, is this a more humane way to keep dogs from destroying things? (opposed to keep them outside or in a cage). Or, are we creating obsessive compulsive animals that require constant feedback? Are dogs even interested in an interactive experience?
I’ll put this concept (Dog Gamification) in consideration for a prototype build. I might be able to assemble a very primitive version of it with some Arduinos and Raspberry Pis and some cheap sensors that I find online. That will be part 2 : Stay tuned.
Let’s say you have a brand new idea. Let’s be honest, maybe you should just forget about it… Why? Because by definition you are absolutely biased by your own background, assumptions and context. You are to such idea what a mom is to her baby. You won’t see anything wrong with it, you’ll love it and you’ll do anything it takes to demonstrate the world that you are right (even if you aren’t).
As I was doing the laundry last Sunday, I realized that not because I do it every week, the clothes are getting cleaner, they are just the same every week. Same as the Jackpot player that pulls the lever once or a thousand times, the probabilities don’t really increase as it resets every time however subconsciously there is indeed a sense of progress. How many hours have you spent thinking about that awesome idea you had a year ago? How do you know that every day you don’t just reset your vision and replay the same scenarios on how that idea will succeed?
In order to break that mental ping pong you need to run a very simple mental exercise:
“Assume that exactly the same idea was already imagined by somebody smarter that you and with more funding than you… but it failed.”
By doing so, the following happens :
You stop day-dreaming that you have the killer, world changing idea that will be sold out the first day and focus in what is important which is your product.
It brings you closer to the hard reality that you might fail as well but it also gives you the chance to demonstrate that your execution and tactics are much better.
When somebody gives you feedback, you’ll actually listen instead of trying to “correct” them as they could be giving you valuable information that the other guys might have ignored.
You become more cold blooded with trivial features as making a series of wrong choices could have been the reason of the other guy’s demise .
You stop pretending you can be everything for everybody. You want to live to see the next day even if that involves only serving one segment at a time.
As the saying goes: “It is not the idea, it is the execution what matters”. Execution is the art of smuggling an idea from the Imaginary world to the Real world. The crystallization of something ethereal. We should focus on being the Marco Polo of ideas, you and only you know your silk road.
Every time I meet a film director I ask them the same question: If I gave the same script and budget to 5 different directors… would I get the same movie? The answer is unequivocally “no”.
So, why ideas would be different? Ideas being the equivalent of scripts. I would bet you would not be able to replicate Facebook even if you were sent back in time to that dorm at that moment in history same as you wouldn’t be able to replicate Clockwork Orange by Stanley Kubrick even if you had the exact same crew and script.
That is actually good news. Because you could be the “One” that executes right, that non original idea you just got. : )
One thing I know for sure… in my next life I’m going to be either a mouse, a rat or a roach. The reason I know that for sure is because I’ve killed so many since I’ve moved to Brooklyn that I have perfected my technique in ways I almost want to write a handbook about: Mice have really good hearing but they don’t have good eyes. If you stay still long enough they’ll start coming out from the corners. Or Rats, they are just messy and loud. They’ll just chomp through a box instead of trying to open it. Rats are zero stealth but they are damn fast. They can jump to places you can’t imagine (like to the top of a table) and eat through a wall. Roaches on the other hand, they basically hate humans, they go out when you are not there and pretend they are dead when you look at them.
So, what we know is that plagues: a. follow certain patterns , b. You’ll never see them enjoying themselves because they are always running from you. For every time you don’t see them , they are 100x on your food, on your clothes, on your dishes, etc. Maybe you just don’t want to know and pretend they don’t exist… or, you can use technology to reveal what really is going on =).
Plan of Action
We are going to design an hypothetical gadget that will capture images of the mice, rats or roaches while they are roaming around looking for food or just hanging out in an open space. This gadget would be a home version of the movement triggered cameras that nature photographers use to capture wildlife.
Gadget is placed on floor level pointing to a corner or hole were the animal or insect is suspected to come from. Optionally, it could be pointing to a bait that we have setup for this purpose.
Camera sends frames at a constant rate to the Stick GPU running the program
Frame is input to the Movement detection neural network
If Movement is detected, Frame is input to the Plague detection neural network
If a rat, mouse or roach is detected, image is stored in Flash Drive.
Once every hour, images are posted to IFTTT to a rule that send them to your email.
Have the program remember if is there have been recent events and have it in state of alert
Infer the presence of the Rat,Mouse or Roach from multiple frames (instead of one frame).
Use a microphone to cross reference sounds with images.
+ Even though a GPU Stick is the best option, a Raspberry Pi 3 could be capable to do the job. I’ve successfully run a face detection NN, a face recognition NN and a QR Code reader NN on a Raspberry Pi 3 in other projects.
+ No NN training is assumed.
+ I’ve never seen a Plague detection NN out there, it is just hypothetical. There is a big possibility that it is not even possible to build because of light conditions and size and speed of mice and rats.
+ The Internet Connection to IFTT is entirely optional. The user could retrieve the images from the SD Card directly.
There is a very curious thing that happens when you are trying to learn a new (spoken) language, you run out of words pretty fast. When I was just learning english I wanted to use the work “good” for practically everything. Food was “good”, a movie was “good”, my seat was “good” and generally speaking life was “good”. People would definitely understand me but I felt kind of trapped in this one dimensional linguistic space. Problem is, that ideas are everything but one dimension.
Transplanting that concept to Smart Home, I was originally planning to write this blog under the constraints that every proposed idea was to be built with existing plug and play components available in the market. However… I found myself trying to work around the limited catalogue of existing components and basically coming up with one dimensional ideas. On the other hand, I could put my hat of electronic engineer, grab my multimeter and my soldering iron and build them myself with resistors, micro controllers, etc, however that would be like cheating, and I would be violating the principle of “The skill to setup a zombie home should be equivalent to the skills to connect any other home appliance.” .
For that reason, I’m going to propose here a list of elements that will fill an hypothetical box of components that I’ll use in every project.
Assume that each one of these components can just talk to each other. A way to visualize their communication is to see them as actors in a play and they are following a script. Let’s say I want to create a gadget that identifies bad smells in the kitchen.
The play would go like this:
Every five minutes, Sarah “The Controller” listens to Joe “The Air Quality sensor” yelling out loud his readings. When Sarah hears that the air quality is not good, she suggests Bill “The Fan” to start and run until further notice. In the meantime, Sarah suggests Mary “The color cube” to turn red. In addition to that, she writes in a notebook (a flash memory) what just happened and goes to sleep. Five minutes after, Sarah wakes up and pays attention to what Joe (the Air Quality guy) is saying again and figures out if something has changed. It the readings look better now, she suggests Bill (the fan) to stop and Mary (the color cube) to turn green. Sarah goes to sleep again. She repeats this action every 5 minutes.
You might be thinking this is stupid and childish… maybe =), but I’ve designed quite a couple software components in my life and realized early on that when you turn all the components into people, your social skills kick in and you start seeing what everybody is going and not going to do, who has enough information, who are you using a lot, who is being ignored, etc. The result is a much more well though and rounded up piece of software. Just for the sake of simplicity, I’m going to use this paradigm to explain the algorithms that run each one of the gadgets. You just need to assume that such algorithms (or plays) are already loaded in the gadget or in the controller so you don’t need to program them.
Conclusion: I just established a series of hypothetical components that by act of magic just talk to each other as if they were just people. Instead of software we’ll use some sort of play script that describes their interaction.
Today’s (2019) Smart Home offering is very politically correct, every gadget needs to look as if it was “Designed in California”. It needs to blend perfectly with your furniture and it must offer “analytics” not to mention an App in you smartphone.
I get it’s super exciting to see the temperature sway in your living room in real time while you are bored at the office and that the reported savings your smart appliance provided with its smart recommendations (which are partially true) will be a great conversation point, I know you’ll feel good about that light automatically changing its color when you come home after spending more on that bulb than you’d like to admit but… seriously, we can do better than that.
The current Problem : The common assumption is that throwing some sensors and actuators plus a couple of “ideas” to users, we’ll come up with awesome use cases and by doing so the revolution of smart homes will be bootstrapped.
In theory it is not a bad idea, I’m sure other industries have pulled a similar trick, the issue is that it is marketed as a plug and play solution. A real plug and play looks like a Gaming console that you connect and start playing with immediately after. I know a lot of people that have invested in smart gateways/lights or sensors just to open the box once and then abandon it because there is really no path towards solving a particular problem. To put it in perspective, imagine that IKEA sold exclusively a box with all the joints and screws and legs to assemble “something” for your house. I guess you could build the chair of your dreams but that is far from a straight path to solve a problem that you might have today and right now.
My proposal is to focus on problems or at least interesting projects first, technology second. As an engineer this is kind of saying that I’m going to the backseat. As a product designer I say: let’s come up with one hundred awesome ideas that demonstrate that you can have awesome home gadgets (or I’ll eat my hat). And just then, let’s start talking about implementation. If future doesn’t come, let’s bring it in =).
Post apocalyptic movies depict a world were there is no internet, no electric grid, water is not always safe to drink and even more trivial aspects like lighting or communication can never be taken for granted.
You don’t need to go to the future or wait for the zombies to come, just go to a rural area around where you live and you’ll realize we are designing smart home technology for a very specific set of assumptions: Permanent connection to the internet, a smartphone and un-interrupted electric service.
I’m not proposing to cut the cord from the internet, go back to punchcards or operate everything on solar (although that would be super cool). Instead, I want to turn those “Must haves” into “Nice to haves”. Instead of trying to connect that lamp to the internet, let’s focus on things that are more important like: Reusability, Stand Alone capabilities, Simplicity and Cost. The how to get there is the Mission of this Blog.
Zombie Home Manifest
A Zombie Home uses technology to solve real problems, automate boring tasks and provide useful insight but:
A Zombie Home should not depend on smart phones to be operated.
A Zombie Home should not depend on Cloud Services or permanent Internet connection to work.
A Zombie Home should not depend on an electric grid to work.
The technology that makes a Zombie Home gadget work should be 100% salvageable to be used in another gadget.
The skill to setup a zombie home should be equivalent to the skills to connect any other home appliance.
Zombie gadgets should take advantage of old and latest technology without committing to any in particular.
But more important than anything: It should be affordable. Not only affluent people have problems that can be solved in a smart way with technology.