Streaming data processing systems - part 1

Problem: I have a smart environment monitoring device at home, this has a few sensors, like a temperature sensor, a humidity sensor, a carbon monoxide sensor, noise level and air quality sensor, it is also connected to the internet and every 30 seconds, it sends this data to the server. In addition to environment data, it occasionally also sends the device status data every hour, which looks at the overall condition of the device.    

This device has become very popular and now has been in the market for some time, it has been sold to a million plus households. Yes, this means that we have a problem, a good problem statement, which requires us to process all this information at the server. 

This is a typical problem for a stream processing data system to handle.  This is a high velocity and high volume data which needs to be handled by our data processing system.

The Kappa architecture

An architectural pattern which handles a stream processing system is the kappa architecture, which treats streams as first class citizen, apache spark streaming is a framework which has a concept of micro batches and structured streaming to be able to process data from stream oriented systems.

In part two we will take a look at the kappa architecture in detail and come up with an initial design of a system that handles such a use case. 

Low Carb Diet

Following a new low carb diet, high fat, keto diet , apparently carbs especially simple carbs can make you fat and create high insulin levels in the body, which inhibit fat loss.

Following are the high level categories of foods, each category has foods that are allowed and foods that are strictly avoid

1. Vegetables: Anything grown above the ground is generally considered a good low carb vegetable, the following are the most desirable
1. Kale
2. Broccoli raab
3. Watercress
4. Spinach
5. Green leaf lettuce
6. Celery
7. Tomato
8. Cucumber
9. Bok choy
10. White mushrooms
11. Eggplant
12. Asparagus
13. Zucchini
14. Bell peppers
15. Cabbage
16. Cauliflower
17. Brocolli
18. Fennel
19. Green beans
2. Fruits: Fruits with buttery fat and less sweetness are a good choice as they contain less carb
1. Avacado
2. Star fruit
3. Blackberry
4. Raspberry
5. Honeydew melon
6. Strawberry
7. Cantaloupe
8. Lemon
9. Gooseberry
10. Watermelon
11. Peach
12. Apricot
13. Plum
14. Blueberry
3. Nuts: Again, nuts with good fats and fewer carbs
1. Pecan
2. Brazil Nut
3. Macadamia
4. Walnut
5. Hazelnut
6. Peanut
7. Almond
4. Dairy: High fat dairy products are good, avoid milk
1. Butter
2. Cottage Cheese
3. Ghee
4. Cheddar Cheese
5. Fish, Meat and Poultry: Most are good,
1. Fatty fish like, tuna, salmon, sardines
2. Eggs
3. Organic Chicken

Besides the above, special foods that need to be incorporated

• Chia seeds
• Flax Seeds
• Seaweed snacks
• Coconut Oil
• Avacado Oil
• Oat milk