10 January 2023

What are some problems that NiFi solves?

Apache NiFi has been used to solve a wide variety of data integration and data management problems, some of which include:

  1. Data Ingestion: NiFi can be used to collect, acquire and ingest data from a wide variety of sources, such as log files, social media feeds, sensor data, and databases, among others. It supports a wide range of protocols and data formats, which makes it easy to connect to various systems and collect data from them.
  2. Data Flow and Routing: NiFi provides a powerful and flexible data flow and routing engine that allows users to route and process data based on specific conditions. This makes it easy to filter, transform, and route data to different destinations, such as a data lake, a data warehouse, or a real-time analytics system.
  3. Real-time Data Processing: NiFi’s ability to handle data in real time and its low-latency data processing make it well-suited for real-time data processing use cases such as IoT, streaming data, and event-driven architectures. It can process and route large amounts of data in near real time.
  4. Data Governance: NiFi’s security and monitoring features provide visibility into data flows and help organizations ensure data governance, compliance and security.
  5. Edge Processing: NiFi can also be used to perform edge processing, allowing devices and systems at the “edge” of the network to process and filter data before it is sent to a central location. This can help to reduce the amount of data sent over the network and make data processing more efficient.
  6. Data Quality: NiFi can be used to validate, cleanse and enrich data before it is sent to a data lake or data warehouse. This can help organizations to ensure data quality and make the data more useful for analytics and reporting.
  7. Big Data Integration: NiFi can be integrated with other big data tools such as Apache Hadoop, Apache Spark, Apache Kafka, and Apache HBase, allowing organizations to easily move, process, and manage large amounts of data.

These are just a few examples of the many problems that NiFi can be used to solve. Because of its flexibility, scalability, and compatibility with other systems, it can be tailored to a wide variety of data integration and data management use cases.

Category: NiFi | LEAVE A COMMENT
9 January 2023

What is NiFi?

Apache NiFi is an open-source data integration and data flow tool that helps users to move, process, and manage data between different systems. It provides an easy-to-use, web-based interface for designing data flows, and it can be used to collect, route, transform, and process data in real time.

In advanced terms, NiFi is a data integration and data flow tool that provides a web-based interface for designing data flows and managing the flow of data between different systems. It utilizes a directed acyclic graph (DAG) model for data flow, where each node in the graph represents a specific data processing operation, and the edges between the nodes represent the flow of data.

The core concept of NiFi is the flow of data, which is represented by dataflows. The data flows through NiFi by being passed from one Processor (a NiFi component that performs a specific action on the data) to another. NiFi provides a library of pre-built “Processors” that can be used to perform various operations on data, such as collecting data from a source, routing data based on specific conditions, transforming data to a different format, and enriching data with additional information. Additionally, users can create custom processors to perform specialized or proprietary operations on data. The data can also be routed to different destinations, such as a file system, a database, or another system.

NiFi is highly configurable and can be used to solve a wide variety of data integration and management problems. One of its key features is the ability to handle data in real time, which makes it a good fit for use cases such as data streaming and IoT. It provides low-latency data processing and is designed to handle high-volume, high-velocity data streams. NiFi’s architecture is based on a distributed and scalable model, which allows it to easily handle large amounts of data and provides built-in fault tolerance and load balancing.

NiFi also has built-in security features, including user authentication, data encryption in transit and at rest, and access controls. Additionally, it provides monitoring and management features that allow users to track the performance of data flows and troubleshoot issues. These features make it easy to secure data in transit and at rest, and to monitor the performance of data flows. As well as understanding the behavior of data flows, identifying bottlenecks, and finding opportunities for optimization. Additionally, it has an extensible architecture that allows the addition of custom processors which makes it compatible with different systems.

Overall, NiFi is a powerful tool for managing and processing data, and it’s a great choice for organizations that need to integrate data from multiple sources and move it to where it’s needed in real-time.

Category: NiFi | LEAVE A COMMENT
19 November 2022

Using a personalized domain with O365 Family

I want to start this post with the mention that Microsoft is going to be sunsetting this feature at the end of November 2023. So if this is something you are interested in… Hurry up as the opportunity is going away very soon. As per the Microsoft Support page (link): 

“Starting November 30, 2023, Microsoft 365 Personal or Microsoft 365 Family subscribers will no longer be able to create a new personalized email address associated with their Outlook.com inbox,”

Associating and using a personal domain with my O365 Family plan to allow personalized email addresses is something that has really appealed to me, being that I live and work in the tech realm. However, to my disappointment, this feature is only officially supported by Microsoft thru GoDaddy.

I personally do not want to have to switch registrars though. I’m quite happy with the folks that host my name servers and I will not budge on that. GoDaddy might be “ok” at best for tech newb with zero experience. I personally have had bad experiences with them, and their constant attempts of upselling you at every click I find extremely annoying. So then I was left trying to figure out how to still get my domain without switching registrars. Thankfully, thanks to the Internet, i was able to discover that someone already solved my problem. I found a post on Reddit by u/khatarian that explained what I needed to do.

Reminder: While this does work, it is not officially supported.

1) Open Outlook.com in your browser, open ‘Settings’ and go to the ‘Premium’ settings.
Click the “Get started” button under ‘Personalized email address’.

2) Click the button “Get a domain”. A new window will open.

3) Copy the URL of the new window that opens, it will appear like: https://domainconnect.godaddy.com/v2/domainTemplates/providers/outlook.com/services/personalizedoutlookemail/apply?mxRecordValue=XXXXXXXXX&state=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.

The part that is needed is “mxRecordValue=XXXXXXXXX” The XXXXXXXXX is a unique ID that Microsoft will check for during DNS verification. Save that in notepad for later…

4) Go to your own personal site’s DNS settings and add the following values, replacing XXXXXXXXX with the value you retrieved in step 3.
Note: Your DNS manager may use different syntax for it’s entries. Please adapt these entries accordingly. The dot at the end is not a typo!

NameTTLTypeValue
@1 hourMX0 XXXXXXXXX.pamx1.hotmail.com.
@1 hourTXTv=spf1 include:outlook.com -all
autodiscover1 hourCNAMEautodiscover.outlook.com.
_dmarc1 hourTXTv=DMARC1; p=none;
_domainconnect1 hourCNAME_domainconnect.gd.domaincontrol.com.
_outlook1 hourTXTXXXXXXXXX

FWIW – This is what the settings look like at NameCheap.com.

5) Return to your ‘Premium Settings’ in Outlook.com

6) Select and click the option “I already own a GoDaddy doamin”

7) Enter the personalized domain name that you used above and have configured the DNS entries for.

8) Click the “Sign in” button and then close the page it opens.

9) Woo-Hoo! You did it. It should now say that you are connected to the domain.

I’m confirmed that it works for one of the domains I own. I couldn’t be happier. Now it’s time for you to do it before the end of November 2023!

Category: Microsoft, Office | LEAVE A COMMENT
15 November 2022

Helium Token Related Links

Just wanted to bookmark a few links to resources that are related to Helium, specifically Helium miners.

Helium Explorer – https://explorer.helium.com/
Helium Explorer is a Block Explorer and Analytics Platform for Helium

HotSpotty – https://app.hotspotty.net/hotspots/
A mapping tool that helps you visualize the miners on a map and it pulls up some statistics on the miner you click on.

CrowdSpot – https://www.crowdspot.io/hotspots
This tool allows anyone to explore Helium Network Hotspot data in an effort to understand atypical Hotspot behavior and identify anomalies.

Category: Crypto-Things, Helium | LEAVE A COMMENT
26 September 2022

Microsoft | Learn

I decided to write a little bit about the Microsoft Learn portal. We all know that Microsoft is one (if not the) standard for many business applications and cloud offerings. They also offer certifications on their products from basic fundamentals all the way up to the expert level. But how do you learn, practice, and eventually master the skills/knowledge needed to either professionally use or become certified in any of the Microsoft family of technologies? Well, the answer is Microsoft Learn.

The Microsoft Learn portal can really meet all of your learning needs. It can provide you the documentation for any of their current product offerings. It can walk you through any of the Microsoft exams, and the skills they cover. Microsoft has put together self-paced training modules that you can use. And best of all, they basically offer some sort of module for everything they offer. If you are interested in learning anything Microsoft-related, then it really behooves you to start looking at the Microsoft Learn site to begin or supplement your learning materials.

There is so much knowledge that is just sitting there, waiting for you. If it’s a Microsoft product, and you’re interested in it, then Microsoft Learn is where you need to go!

Category: Microsoft, Microsoft, Training | LEAVE A COMMENT
26 July 2022

Docker Commands Cheat Sheet

run container

docker run

run container and drop to shell

docker run -it 
docker run --interactive --tty

run container in background 

docker run -d

container restart setings

docker run --restart (always|no|on-failure[:maxretries]|unless-stopped)

remove container when exited

docker run --rm

view containers running

docker container ls
docker container ls -a

start container

docker start

stop container

docker stop

remove container

docker rm

remove all stopped

docker container prune

rename container

docker rename

view containers metrics

docker stats []

copy file into container

docker cp file.ext :/path/to/folder/

view container information

docker inspect

Find container’s IP

docker inspect | grep IPAddress

Make a container an image

docker commit

Map container port to host port

docker run -p :
Category: Containers, Docker | LEAVE A COMMENT
24 July 2022

Install Docker CE on Amazon Linux 2

Here is how to install docker and docker-compose on the (AWS) Amazon Linux 2 OS running on either an EC2 or Lightsail instance.

Let’s begin by opening a console or SSH session to your EC2 or Lightsail instance. You can do this from within your AWS portal or an SSH tool like Putty. The actual “how to connect” to your server is outside the scope of this article.

Once you are connected, let us start by installing any pending updates on your host. 

sudo yum update

Next, we will install Docker. 

sudo yum install docker

Create a new membership group for docker and add the ec2-user to it so you can run all of the docker commands without needing to use the sudo command.

sudo usermod -a -G docker ec2-user
id ec2-user
newgrp docker

Now it’s time to add docker-compose.

wget https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) 
sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/local/bin/docker-compose
sudo chmod -v +x /usr/local/bin/docker-compose

Enable the docker service.

sudo systemctl enable docker.service

Start the docker service.

sudo systemctl start docker.service

Verify that the service is running.

sudo systemctl status docker.service

You can check the docker version with this command.

docker version

And lastly you can check the docker-compose version with this command..

docker-compose version

You have now installed docker and docker-compose on Amazon Linux 2.
I am going to leave off with some helpful controls manage docker on your machine.

sudo systemctl start docker.service #start docker
sudo systemctl stop docker.service #stop docker
sudo systemctl restart docker.service #restart docker
sudo systemctl status docker.service #get the status of docker
Category: AWS, Cloud, Containers, Docker, Linux | LEAVE A COMMENT
24 July 2022

Install Docker CE on Ubuntu

So how does someone install Docker on Ubuntu? Let me show you… We’re starting off with a freshly installed Ubuntu 20.04 virtual machine that has been updated but has not had anything additional added to it yet.

We will start with adding some packages that are prerequisites for using the ‘apt’ commands over HTTPS and thus for Docker.

sudo apt install apt-transport-https ca-certificates curl software-properties-common

Next, we’ll add the GPG key for the Official Docker repository.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Then we’ll add the Docker repo to the sources for APT.

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

We’ll rerun our ‘update’ command.

sudo apt update

And the step everyone was waiting for, installing Docker.

sudo apt install docker-ce

Now we need to allow our user to run Docker commands without always needing to ‘sudo’. We can add the user we are logged in as by using this command.

sudo usermod -aG docker ${USER}

Okay now as an optional step, you can install Docker-Compose. At the time of writing this, it is on v2.7.0. You will want to check their release page and update the command below to the current version number.

mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.7.0/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose

Then we can set the correct permissions on docker-compose so that it will be executable. 

chmod +x ~/.docker/cli-plugins/docker-compose

Now we can verify our Docker and Docker-Compose are installed by checking their versions with these commands. 

docker compose version
docker version

And that is how you install Docker on your Ubuntu machine

Category: Containers, Docker, Linux, Ubuntu | LEAVE A COMMENT
10 July 2022

Install Git on CentOS

Okay, it’s time to install Git so you can play with some pull and merge requests for some projects you are working on. This set of instructions should work on your system regardless if you are running CentOS 7, 8, or 9.

The first thing to do is elevate…

sudo su

Then update your system.

dnf update

Install Git

dnf install git

Check the installed version

git --version

Just like that, you are ready to “Git” yourself back to coding something grand!

Category: CentOS, Git | LEAVE A COMMENT
9 July 2022

Install Docker CE on CentOS 9 Stream

Docker is an operating system virtualization tool that allows us to run applications as containers. In simplest terms, that means you are virtualizing only the application, and not creating an entire virtual machine as you would traditionally do in hypervisors like VMware, Hyper-V, or Nutanix.

Okay, that’s cool… How do we install Docker so we can start to test workloads on it? Well, let me show you how to install Docker on a virtual machine running CentOS 9 Stream.
**While I have not tested to confirm, this Docker installation method should be identical on CentOS 8 Stream, as well as for CentOS 7.x

Let us begin by shifting to Sudo mode by running this command first…

sudo su

Then the first thing to do is remove PodMan as it conflicts with Docker.

dnf -y remove podman runc

The next step is to add the Docker repo.

curl https://download.docker.com/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo

Update SELinux in regards to the Docker repo.

sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/docker-ce.repo

Enable the Docker repo and install Docker.

dnf --enablerepo=docker-ce-stable -y install docker-ce

With Docker installed, it is time to enable it.

systemctl start docker
systemctl enable docker

Let us view what we installed by running these two commands.

rpm -q docker-ce
docker version

Congratulations! You now have Docker installed on your machine.

You’ll probably want to install Docker Compose on your machine too so you can build and run a docker image. You can install it with this simple command.

dnf install docker-compose
Category: CentOS, Containers, Docker, Linux | LEAVE A COMMENT