Rakesh M
Author Archives: Rakesh M
- Home → Author's archive:
Rakesh M
Opensource LLM Models – Meta llama / Meta Codellama ? Deploying In-house ? Context of Networking!
Disclaimer: All Writings And Opinions Are My Own And Are Interpreted Solely From My Understanding. Please Contact The Concerned Support Teams For A Professional Opinion, As Technology And Features Change Rapidly.
In a world where even your toaster might soon have a PhD in quantum physics, LLMs are taking over faster than a cat video going viral! LLMs are becoming increasingly powerful and are being integrated into various business and personal use cases. Networking is no different. Due to reasons like privacy, connectivity, and cost, deploying smaller form factor models or larger ones (if you can afford in-house compute) is becoming more feasible for faster inference and lower cost.
The availability and cost of model inference are improving rapidly. While OpenAI’s ChatGPT-4 is well-known, Meta and other firms are also developing LLMs that can be deployed in-house and fine-tuned for various scenarios.
Let’s explore how to deploy an open-source model in the context of coding. For beginners, ease of deployment is crucial; nothing is more off-putting than a complicated setup.
Reference : Ollama.com (https://github.com/ollama/ollama?tab=readme-ov-file) simplifies fetching a model and starting work immediately.
Visit ollama.com to understand what a codellama model looks like and what Continue reading
Oracle OCI Traffic Mirroring and Monitoring: VTAP Implementation and Analysis
Disclaimer: All writings and opinions are my own and are interpreted solely from my understanding. Please contact the concerned support teams for a professional opinion, as technology and features change rapidly.
My name is Stephen King, and you are reading my novel. Absolutely Not! He is the most incredible author of all time! And you are reading my blog! One of my many, many, many interests is traffic mirroring and monitoring in public clouds, especially inter-VCN/VPC traffic. Traffic from an instance is mirrored and sent for any analysis, whether regulatory or troubleshooting. I quickly set up something in my OCI; the results and learnings are fascinating.
TLDR: Traffic Mirroring and Monitoring in Oracle OCI using VTAPs
Topology and a refresher
IGW helps us connect to the Internet, NLB helps us send traffic to VTAP-HOST mirrored from VTAP, and a DRG helps us communicate with other VCNs.
What is the end goal? Mirror and send all the traffic from Host-1 with IP 192.168.1.6 to VTAP-Host for further analysis.
Below is generated by OCI Network Visualiser, which is very cool.
A few things Continue reading
Oracle OCI DRCC – Who, What, How? Regional Subnet – Happiness
Disclaimer: All writings and opinions are my own and are interpreted solely from my understanding. Please contact the concerned support teams for a professional opinion, as technology and features change rapidly.
Topics Discussed: DRCCs Intro and OCI Cloud Regional Subnet
Spoiler alert: This involves an on-premises cloud for customers offered by Oracle. All offerings for cloud@customer by Oracle can be viewed at https://www.oracle.com/cloud/cloud-at-customer/
Dedicate Region Cloud @Customer – DRCC
Ref: https://www.oracle.com/cloud/cloud-at-customer/dedicated-region/#rc30category-documentation
Before understanding DRCC, let’s consider the scope of improvement on customer premises with their own Data Center and what it tries to solve. Fair? Let’s imagine I have my own data centre. Great! My infrastructure supports some of my workloads, but how do I get the latest technology stack, support, and services that the public cloud offers?
For example, how can someone leverage all these services without having to invest in development cycles yet meet latency/integration/data security and multiple other requirements?
Ref: https://www.oracle.com/cloud/
| Scope of improvement on customer premises | Oracle Dedicated Region Cloud at Customer Solutions |
| Data Security and Compliance Concerns | Provides a fully managed cloud region in the customer’s datacenter, ensuring data sovereignty and compliance with local regulations. |
| Latency Continue reading |
Energy Independence, at least for gadgets and electronics; what’s the point of growing spinach and Mint?
This post is a bit different from my usual networking content. I won’t be discussing power grid inefficiencies or pollution. After learning some principles from permaculture, an ethical design science focused on life, it’s more of a personal reflection. I’m concerned about our society’s reliance on non-producing, ever-consuming and our mindset to depend on external producers. Even simple things that can be eliminated or produced without much effort have become needlessly complex, creating unnecessary consumption, erasing the human spark to make or say no to mindless consumerism, and creating vast dump yards. We used to be producers but have become ultimate consumers of life’s simple things.
TLDR: It’s not about affordability or money; I think everyone here can afford spinach or charge their devices happily, paying a service provider; the question is, why are we dependent on external grids and super-stores for simple things in life that were free at one point of time in our society, over consumerism, isolated and end up creating our own silo kingdoms?
Why did it make sense to me?
At a very high level, we all know that the climatic changes and food production, in general, have affected the topsoil so much that Continue reading
A While, Career update and Ends with various Junos load commands.
It has been quite some time since my last blog post. The past few months have been busy, leaving little time to write. I am happy to share that I have started working as a site-reliability developer at Oracle. While it has only been a short while, I am enjoying the work.
Reflecting on the past year, I am happy to say it has been productive overall. I have had the opportunity to collaborate with various networking and software teams, which has taught me a lot about high-scale traffic patterns and migrations. Working in an operations role has been beneficial since it requires constant fire-fighting and documentation for improvements. This has made me more aware of traffic patterns and ways to improve reliability.
What can be improved?
However, I could benefit from more study on the software architecture and distributed system design parameters from a business perspective. I want to write more this year; the first half was better, but lacked in the second half, where most of the writing was private.
From a personal standpoint, 2024 needs more travel absolutely and some improvements in garden automation for summer 2024
Other updates
Finally, I have been working on some JNCIA-Devops Continue reading
AWS Advanced Networking Speciality 1.3:Considerations for encryption and authentication with load balancers (for example, TLStermination, TLS passthrough)
< MEDIUM :https://towardsaws.com/aws-advanced-networking-speciality-1-3-considerations-402e0d057dfb >
List of blogs on AWS Advanced Networking Speciality Exam — https://medium.com/@raaki-88/list/aws-advanced-network-speciality-24009c3d8474
Data Protection in ELB
AWS Shared-Responsibility Model defines how data protection applies in ELBs. It boils down to AWS protecting global infrastructure while the service consumer is more responsible for preserving the content and control over the hosted content.
Few important suggestions for accessing/Securing
- MFA for accounts
- TLS 1.2 or TLS 1.3 for AWS resource communication
- Logging with AWS CloudTrail
- Amazon Macie — Discovering and securing sensitive data in S3
- FIP140–2 — Fips Endpoint
Encryption
Encryption at rest: Server-side encryption for S3 (SSE-S3) is used for ELB access logs. ELB automatically encrypts each log file before storing it in the S3 bucket and decrypts the access log files when you access them. Each log file is encrypted with a unique key, which is encrypted with a master key that is regularly rotated.
Encryption in Transit:
HTTPS/TLS traffic can be terminated at the ELB. ELB can encrypt and decrypt the traffic instead of additional EC2 instances or current EC2 backend instances doing this TLS termination. Using ACM (AWS Certificate Continue reading
AWS Advanced Networking Speciality 1.3: AWS Load Balancer Controller for Kubernetes clusters
List of blogs on AWS Advanced Networking Speciality Exam — https://medium.com/@raaki-88/list/aws-advanced-network-speciality-24009c3d8474
Before understanding LoadBalancer Service, it’s worth understanding a few things about NodePort service.
NodePort Service :
NodePort service opens a port on each node. External agents can connect directly to the node on the NodePort. If not mentioned, a randomly chosen service is picked up for NodePort. LoadBalancing here is managed by front-end service, which listens to a port and load balances across the Pods, which responds to service requests.
LoadBalancer Service:
Like NodePort Service, the LoadBalancer service extends the functionality by adding a load balancer in front of all the nodes. Kubernetes requests ELB and registers all the nodes. It’s worth noting that Load Balancer will not detect where the pods are running. Worker nodes are added as backend instances in the load balancer. The classic-load balancer is the default LB the service chooses and can be changed to NLB(Network Load Balancer). CLB routes the requests to Front-end, then to internal service ports Continue reading
AWS Advanced Networking Speciality 1.3 : Configuration options for load balancer target groups (for example, TCP, GENEVE, IP compared with instance)
Advanced Network Speciality Exam — Blogs
<MEDIUM : https://towardsaws.com/aws-advanced-networking-speciality-1-3-5484de6c8da >
A Target group routes requests to one or more registered targets. They can be EC2 Instances, IP addresses, Kubernetes Cluster, Lambda Functions etc. Target groups are specified when you create a listener rule. You can also define various health checks and associate them with each target-groups.
Typical load-balancer components
What is Geneve, and what is the context with ELB: Generic Network Virtualisation Encapsulation
In the context of Gateway Load Balancer, a flow can be associated with either 5-Tuple or 3-Tuple.A flow can be associated with either a 5-tuple or 3-tuple flow in load balancers.
A 5-tuple flow includes the source IP address, destination IP address, source port, destination port, and protocol number. This is used for TCP, UDP, and SCTP protocols.
A 3-tuple flow includes the source IP address, destination IP address, and protocol number. This is used for ICMP and ICMPv6 protocols.
Gateway Load balancers and their registered virtual appliances use GENEVE protocol to exchange application traffic on port 6081
References :
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html
https://datatracker.ietf.org/doc/html/rfc8926 Continue reading
Speciality 1.3 : Integrations of load balancers and other AWS services (for example, Global Accelerator, CloudFront, AWS WAF, Route 53, Amazon Elastic Kubernetes Service [Amazon EKS], AWS Certificate Manager [ACM])
< MEDIUM : https://raaki-88.medium.com/aws-advanced-networking-speciality-1-3-deedc0217ea6 >
Advanced Network Speciality Exam — Blogs
https://medium.com/@raaki-88/list/aws-advanced-network-speciality-24009c3d8474
Global Accelerator — A service that provides static ip addresses with your accelerator. These IP addresses are Anycast from the AWS edge network, meaning the global accelerator diverts your application’s traffic to the nearest region to the client.
Two types of Global Accelerators — Standard Accelerators and Custom Routing accelerators.
Standard Accelerators uses aws global network to route traffic to the optimal regional endpoint based on health, client location and policies that the user configures, increasing availability and decreasing latency to the end users. Standard-accelerator endpoints can be Network Load balancers, Application load balancers from load balancing context. Custom routing accelerators do not support load balancer endpoints as of today.
When using accelerators and Load-balancers, update DNS records so that application traffic uses accelerator end-point, redirecting the traffic to load-balancer endpoints.
CloudFront and AWS-WAF with ELB:
When using an application load balancer in ELB, cloud-front meant to cache the objects can reduce the load on ALBs and improve performance. CF can also protect ALB and internal services from DDOS attacks, as with AWS WAF. But for this to succeed, administrators Continue reading
AWS Advanced Networking Speciality 1.3 AWS Advanced Networking: Connectivity patterns that apply to load balancing based on the use case
< MEDIUM: https://raaki-88.medium.com/aws-advanced-networking-speciality-1-3-23eb011b74df >
Previous posts :
https://towardsaws.com/aws-advanced-networking-task-statement-1-3-c457fa0ed893
https://raaki-88.medium.com/aws-advanced-networking-speciality-1-3-3ffe2a43e2f3
Internal ELB — An internal Load balancer is not exposed to the internet and is deployed in a private subnet. A DNS record gets created, which will have a private-IP address of the load-balancer. It’s worth noting to know DNS records will be publicly resolvable. The main intention is to distribute traffic to EC2 instances. Across availability zones, provided all of them have access to VPCs.
External ELB — Also called an Internet-Facing Load Balancer and deployed in the Public subnet. Similar to Internal ELB, this can also be used to distribute and balance traffic across two availability zones.
Example Architecture Reference — https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-amazon-api-gateway-api-on-an-internal-website-using-private-endpoints-and-an-application-load-balancer.html?did=pg_card&trk=pg_card
– Rakesh
AWS Advanced Networking Speciality 1.3 AWS Advanced Networking: Different types of load balancers and how they meet requirements for network design, high availability, and security
< MEDIUM: https://raaki-88.medium.com/aws-advanced-networking-speciality-1-3-3ffe2a43e2f3 >
https://medium.com/towards-aws/aws-advanced-networking-task-statement-1-3-c457fa0ed893 — Has intro details for the speciality exam topic
Different types of Load-Balancers
- CLB — Classic Load Balancer
- ALB — Application Load Balancer
- NLB — Network Load Balancer
- GLB — Gateway Load Balancer
High Availability Aspect: ELB (Can be any load-balancing) can distribute traffic across various different targets, including EC2-Instances, Containers, and IP addresses in either a single AZ or multiple AZs within a region.
Health Checks: An additional health check can be included to ensure that the end hos serving the application is healthy. This is typically done through HTTP status codes, with a return value 200 indicating a healthy host. If discrepancies are found during the health check, the ELB can gracefully drain traffic and divert it to another host in the target group. If auto-scaling is set up, it can also auto-scale as needed.
Network Design: Depending on the type of traffic and Application traffic pattern, the load and burst-rate choice of load-balancer will differ.
Various Features — High-Availability, High-Throughput, Health-Checks, Sticky-Sessions, Operational-Monitoring and Logging, Delete-Protection.
TLS Termination — You can also have integrated certificate management and SSL decryption which offloads end-host CPU load and acts as a central Continue reading
AWS Advanced Networking — Task Statement 1.3: How load balancing works at Layer 3, layer 4, and Layer 7 of the OSI model
< MEDIUM: https://towardsaws.com/aws-advanced-networking-task-statement-1-3-c457fa0ed893 >
Ref: https://aws.amazon.com/elasticloadbalancing/
Why Elastic Load Balancer?
- To Distribute incoming application traffic across EC2 instances / Containers / IP Addresses and these entities, generally, can be called targets and grouped as Target-Groups.
- Another advantage is performing health checks, such as packet-loss or high-latency, and can be integrated with Auto-scaling, hence Elastic.
- Depending on the ELB type and its operational requirement, this can further be subdivided into CLB, ALB, GLB, and NLB.
CLB — Classic Load Balancer
– AWS Recommends ALB today instead of CLB
– Intended for EC2 instances which are built in EC2-Classic Network
– Layer 4 or Layer 7 Load Balancing
– Provides SSL Offloading and IPv6 support for Classic Networks
ALB — Application Load Balancer
- Works at the 7th Layer of the OSI Model.
- Supports applications that run in Containers
- Supports Content-based Routing
- HTTP/HTTPS, Mobile Apps, Containers in EC2, and Microservices benefit greatly from ALB
Why Elastic Load Balancer?
- To Distribute incoming application traffic across EC2 instances / Containers / IP Addresses and these entities, generally, can be called targets and grouped as Target-Groups.
- Another advantage is performing health checks, such as packet-loss or high-latency, and can be integrated with Auto-scaling, Continue reading
Set up your Own wireguard-based Cloud VPN server without installing software on your laptop — GL-A1300 Slate Plus Wireless Router
Note: This requires the purchase of a wireless router which is capable of running a Wireguard package in this case it’s Slate-Plus GL-A1300 and I do not have any affiliate or ads program with them, I simply liked it for its effectiveness and low cost.
The Need :
For one reason many of us want a VPN server which does decent encryption but won’t charge us a lot of money, in some cases, it can be done free of cost and in others for not want us to install a variety of software which messes up with internal client routing and also against some of the IT-Policies, even if it’s a browser-based plugin.
The Choice :
Wireguard: https://www.wireguard.com/ — VPN Software, Software-based encryption, extremely fast and light-weight.
GL-A1300 Slate-Plus — Wireless Router with support for Wireguard which is not a feature in many of the current market routers, had OpenWrt as the installed software.
Features
The GL-A1300 Slate Plus wireless VPN encrypted travel router comes packed with features that will make your life easier while travelling. Here are just a few of the most important:
- VPN Encryption — The GL-A1300 Slate Plus Continue reading
AWS Rekognition — A simple classifier without having to build a lot of in-house or Cloud Environments — Sample Bird-Project
A few weeks ago, I set up a bird feeder and used it to capture bird images, the classifier itself was not that accurate but was doing a decent job, what I have realised is that not every time we end up with highly accurate on-board edge classification especially while learning how to implement them.
So after a few weeks, there were a lot of images some of them sure enough had birds while some of them were taken in Pitch Dark and am not even sure what made the classifier figure out a birdie in the snapshot from the camera.
Now for me in order to make an Image, I have to rely on a re-classifier doing the job for me, initially, I thought I will write a lambda-based classifier as a learning experiment, but then I thought it was a one-time process every 6 months or so, so I went ahead with a managed service option and in this case, its AWS Rekognition, which is quite amazing.
https://aws.amazon.com/rekognition/
Code Snippet used for Classification
import os
import concurrent.futures
import boto3
def detect_birds(image_path):
# Configure AWS credentials and region
session Continue reading
Setting up your own Cloud-GPU Server, Jupyter and Anaconda — Easy and complete walkthrough
Note: One of the important tips for lab environments is to set an auto-shutdown timer, below is one such setting in GCP
I have been working on a few hosted environments which include AWS Sagemaker Notebook instances, Google Cloud Colab, Gradient (Paperspace) etc and all of them are really good and needed monthly subscriptions, I decided to have my own GPU server instance which can be personalized and I get charged on a granular basis.
Installing it is not easy, first, you need to find a cloud-computing instance which has GPU support enabled, AWS and GCP are straightforward in this section as the selection is really easy.
Let’s break this into 3 stages
- Selecting a GPU server-based instance for ML practice.
- Installing Jupyter Server — Pain-Point Making it accessible from the internet.
- Installing Package managers like Anaconda — Pain-Point having Kernel of conda reflect in Jupyter lab.
Stage-1
For a change, I will be using GCP in this case from my usual choice of AWS here.
Choose GPU alongside the Instance
Generic Guidelines — https://cloud.google.com/deep-learning-vm/docs/cloud-marketplace
rakesh@instance-1:~$ sudo apt install jupyter-notebook # Step1: generate the file by typing this line in console jupyter notebook Continue reading
Nvidia Jetson Nano — Initial thoughts, impressions, AI Specialist Certification.
While browsing through various ways to get my AI-enabled bird camera setup, I came across Nvidia Jetson Nano, there are varied versions of this product and availability are limited, I Am in Europe and ordered the Nano Developer kit from Amazon US, shipping was fast with a good amount of inbound tax as well.
https://developer.nvidia.com/embedded/jetson-nano-developer-kit — is the one I have purchased while both new and old versions are available.
Unboxing Video :
Initial Impressions and disadvantages:
- I am surprised that this does not have WIFI and only works on an Ethernet adapter, so I ended up purchasing a Wifi-dongle which operated out of the box, I recommend a TP-Link adapter but anything will work and here is the link — https://www.amazon.co.uk/dp/B07LGMD97Z?psc=1&ref=ppx_yo2ov_dt_b_product_details
- One of the factors is that Nvidia on its website claims that they support Raspberry Pi cameras which I have many and the reality is that they won’t end up supporting any of the latest cameras which are based on the IMX7* series, they can support the IMX219 series if you are lucky but anything other than that is a Continue reading
Image Search with Bing — ML/AI Fast.ai and AWS Sage Maker
< MEDIUM: https://aws.plainenglish.io/image-search-with-bing-ml-ai-fast-ai-and-aws-sage-maker-61fae1647c >
If you have heard about awesome AI course that fast.ai offers which is free of charge then you should definitely checkout. https://course.fast.ai/ is where you will find out all the details.
Course takes a very hands on approach and anyone can write and bring up their ML model in under 2 hours of the course. As a part of any image classification one of the basic requirements is to have number of images which is often referred to as Dataset and this Dataset is split into Test/Train/Validation for the model training.
For some of the easier examples, you can rely on search engine to give you those images for you, previously fast.ai used Microsoft’s bing.com for image search but later they replaced it with DDG (DuckDuckGo). while DDG is really nice I had throttle issues and some of the packages were outdated and hard to read.
So, I have re-written the same image-search Python function which uses Microsoft’s bing.com search engine
What are Pre-req:
- Azure Cloud account
- API Keys for the service
Procedure to generate keys
- Go to Azure, create a Resource
- Search for “Bing Search”
- Select “Bing Continue reading
Automating Green-House Photos through Event-Bridge Pipes and Lambda
Image sent to Telegram
I have a small greenhouse which was in the pipeline for over 2 years and I finally decided to build it. Whoever is in gardening will agree that anything grows better in the greenhouse at least it appears to be so.
Now, the initial impression is all good but I have plans to learn and explore both the plant sides of things and also some using some part of image analysis for a predictive action, for all that to happen I need a camera and a picture to start with.
Hardware —
- Raspberry Pi — I have an old one at home, you can technically have any shape or size as long as it fits your need, My recommendation — is Raspberry Pi Zero
What are the other simplest alternatives:
- I could have written a Python script which directly could have sent the image to Telegram storing the image locally or uploading it to S3
The reason I choose to go with Event-bridge Pipe is to put this more into practice and from there on connect more Lambda and step-functions for future expansion of the project.
Architecture Diagram for sending Images Continue reading
Home Automation — Finally Roller Curtains and Nightmares
< MEDIUM: https://raaki-88.medium.com/home-automation-finally-roller-curtains-and-nightmares-b8ef1fc473d9 >
Am a fan and enthusiast of home automation, tried various things in the past and now settled with few things which I would like to share.
- Light Automation is the first and most popular thing to do — Initially I started wiring with PIR sensor available in Amazon, later upgraded the entire light system to Philips Hue and along with Philips Motion sensor, so far so good. Its expensive than my initial solution but had to choose that because of wired vs wireless situation in home
- Smart-plugs are another common set of devices — while I have both combinations of commercially available one and also I personally flashed few of Sonoff smart switches with Tasmota firmware
. . .
Lets get to the Curtain Rollers — So for these here is the catch, I have a remote for these and thats about it, nothing more nothing less, My ideas were mostly around having network connectivity and manipulating them.
- First and foremost thought that these are blue-tooth based and am wrong
- I went to DFROBOT and bought IR-Transmitter and IR-Receiver, nope they dint work
- I went Amazon and bought some other IR-transmitter and Receiver — Again, wasted a Continue reading
Enabling Nested-Virtualisation on Google cloud platform Instance
< MEDIUM: https://raaki-88.medium.com/enabling-nested-virtualisation-on-google-cloud-platform-instance-7f80f3120834 >
Important Excerpt from the below page.
https://cloud.google.com/compute/docs/instances/nested-virtualization/overview
You must run Linux-based OSes that can run QEMU; you can’t use Windows Server images.
You can’t use E2, N2D, or N1 with attached GPUs, and A2 machine types.
You must use Intel Haswell or later processors; AMD processors are not supported. If the default processor for a zone is Sandy Bridge or Ivy Bridge, change the minimum CPU selection for the VMs in that zone to Intel Haswell or later. For information about the processors supported in each zone, see Available regions and zones.
Though there are many use cases, I will speak from a networking standpoint. Let us say you need to do some sort of lab based on popular vMX Juniper or Cisco or any other vendor, if you have a bare metal instance, you have the ability to access the virtualised CPU cores and allocate them to the Qemu which will be the underlying emulator.
Issue
Almost by default most of the cloud providers will disable access to VT-x because of various reasons and some instances are not capable of supporting this by default. So either choose a custom instance with Continue reading