Its always exciting to self host personal websites, cloud servers, ftp servers, CI servers, SSH etc. from our own PCs, old laptops, or even a Raspberry Pi, so that it gives us the satisfaction of setting up, maintaining and using our own server run from our home, which can be publicly accessed anywhere from the world.
Its as easy as finding a reliable internet connection, a device which can be dedicated as the hardware for the server, and some free-software installations and minor configurations. A Raspberry Pi will satisfy almost all needs of a typical user who wants a reliable home server setup, for web server software, the free and open source Apache server will be more than enough, and finally, to expose our server to the public Internet all we have to do is to configure our router to forward the incoming connections coming to a specific port (which we choose) of our router to the port & IP of the device which we are using as server, this process is called port forwarding. And if we want a host name for our services instead of using our IP address to connect to the server, then we might have to use a dynamic or static DNS provider depending upon whether our IP is dynamic or static. DNS providers assigns our IP with a host name (there are plenty of free and paid DNS providers, for example, https://www.noip.com/). Thats it, our public home server is done.
But a barricade that one may encounter while setting up a publicly accessible home server will be that most Internet Service Providers (ISPs) will be using Carrier-grade NAT (CGNAT) to adjust for the lack of individual IPv4 addresses they have. CGNAT is protocol that work in the 3rd layer (Network Layer) of the OSI model. Simply stated CGNAT will let a single public IP address, A, to be shared by multiple customers of an ISP by creating an internal private network. Each user of the internal private network will have an internal IP, Bx, which is only valid inside the private network. All connections going outside the private network will be sourced from the IP A and a particular port. A reply or incoming connections will be forwared to the internal IP Bx based on the previous outbound phase tracking data. Most simply stated CGNAT is similar to the case when our router connected to another bigger router which in this case is owned and managed by the ISP. So its tedious to do port forwarding twice on our router and the other CGNAT system which is managed by the ISP. So simple port forwarding doesnt help us to expose our server to the Internet if we have CGNAT. One way to avoid CGNAT is ask our ISP for a public, static IP address, where we have a fixed public IP address free from CGNAT, but, it involves lot of paperwork in most countries, is often charged significantly higher than our usual internet plans and even some ISPs provide it only to businesses and not to indivdual users.
A simple way to check if our ISP is using CGNAT is to compare our IP address listed as WAN address by our router with the IP address listed by web services that gives our IP address (for example Google, WhatIsMyIP.com, etc). If both address differ that means that our ISP is using CGNAT, because, the IP address listed by the web services are our real public IP used by CGNAT to communicate with the Internet and the one that we see on our router as WAN address is the private IP offered as part of the CGNAT protocol. Usually in that case the WAN address will be in the 100.64.0.0/10 subnet (from 100.64.0.0 to 100.127.255.255). [refer 1] [refer 2]
So the next question will be how to host the server even with an Internet connection that uses CGNAT. The solution is to use services that setups network tunnels from our server to another public server owned by those services, so that whenever someone tries to access our server using the address provided by the service the service uses a network tunnel to connect the access request to our server. Its more or less similar to the SSH tunneling where we forward a port from the server host to the client host and then to the destination host port. (Remote port forwarding.) refer
One such service is ngrok (secure
introspectable
tunnels to localhost) which offers free and premium tiers services as well as easy to use interface and a
python
wrapper, pyngrok is also
available for
making ngrok readily available from anywhere on the command line and via a convenient Python
API.
ngrok makes services running on ports of our server public using these secure tunnels and
create a
publicly accessible web address to those services. So making our website public is as simple as asking
ngrok to create a tunnel to the port at which our web server is running, ngrok
will create
a web address for us, leading to the website. Simple steps to get started with ngrok can be
found here.
The free tier of ngrok allows to create upto 4 tunnels and 40 connections / minute but the
created
HTTP/TCP tunnels will be on random URLs/ports. Paid / premium tiers offer reserved domains, more tunnels and
connections/minute. If using a free tier the URLs will be different on each tunnel creation, which means
whenever
our server reboots or there is power outage the URLs will be different. And if we are using our server
headless
(without monitor or keyboard) (example Raspberry Pi) then it will be a bit difficult to keep track of the
newly
created URLs on each reboot. One alternative solution to keep track of these URLs is to use a simple python
script
which is able to manage whole ngrok setup and monitoring so that it can update the newly
created URLs
somewhere for example push the URLS to some git repository you own. Also adding those scripts to the crontab reboot
job (or anything similar) of our server can make it run on every boot up of the server. So the script will
initiate
ngrok tunnel creation and update the newly created URLs to somewhere where we can easily access
them.
The following python script which can be added as a crontab @reboot job, will
create
tunnels for services running on predefined ports of our server (for example http [8081] and ssh [22]),
update the
generated public URLs to a git repo, and monitor the ngrok process to initiate a reconnection
if
something goes wrong.
main.py file:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
''' Filename: main.py ''' import time, os from pyngrok import conf, ngrok from script_utils import check_network_connection AUTH_TOKEN_FILE_PATH = "./auth_token.txt" PUBLIC_URL_FILE_PATH = "./p_urls.md" NGROK_SERVER_REGION = "in" # India URL_FILE_HEADER = "## Public URLs:\n\n" # Max 4 tunnels for free plan of ngrok # Dictionary => 'ports': 'protocol' PORTS_TARGETED = { 8081: "http", 22: "tcp" } def _Log__(l_type, message): print("{}: {}".format(l_type, message)) def read_entire_file(f_name): try: with open(f_name) as file_con: data = file_con.read() return data except Exception as e: _Log__("ERR", e) return -1 def set_ngrok_authtoken(): auth_token = read_entire_file(AUTH_TOKEN_FILE_PATH) if not auth_token == -1: final_authtk = auth_token.strip() ngrok.set_auth_token(final_authtk) else: return -1 def get_active_ngrok_tunnels(): return ngrok.get_tunnels() def start_ngrok_tunnel(port, protcol): tunnel = ngrok.connect(port, protcol) return tunnel def kill_ngrok_server(): ngrok.kill() def write_public_urls_to_file(tunnel_data): idx = 1 URL_FILE_CONTENT = URL_FILE_HEADER for tunnel in tunnel_data: URL_FILE_CONTENT = URL_FILE_CONTENT + str(idx) + ". **Protocol:** " + tunnel.proto + \ " | **Public URL:** [" + tunnel.public_url + "](" + tunnel.public_url + ")\n" idx = idx + 1 with open(PUBLIC_URL_FILE_PATH, "w") as op_file: op_file.write(URL_FILE_CONTENT) def update_github_with_url_changes(url_change_id, re_conn_id, reboot_stat): commit_message = "AUTO_COMMIT_URL ==> Reboot: " + str(reboot_stat) + ", Reconnection Indx: " \ + str(re_conn_id) + ", URL Change Indx: " + str(url_change_id) commit_command = 'git commit -am \"' + commit_message + '\"' std_resp = os.popen(commit_command).read() #_Log__("INFO", std_resp) std_resp = os.popen('git push origin master').read() #_Log__("INFO", std_resp) def if_tunnel_changes(cur_tunnels, prev_tunnels): if not len(cur_tunnels) == len(prev_tunnels): return 1 prev_urls = [] for tunnel in prev_tunnels: prev_urls.append(tunnel.public_url) for tunnel in cur_tunnels: if not tunnel.public_url in prev_urls: return 1 return 0 if __name__ == "__main__": is_reboot = True reconnect_indx = 0 while True: connected_to_nw = False url_change_indx = 0 while not connected_to_nw: if check_network_connection() == 1: connected_to_nw = True break time.sleep(1) # retry after 1 sec kill_ngrok_server() set_ngrok_authtoken() conf.get_default().region = NGROK_SERVER_REGION for n_ports, n_protocol in PORTS_TARGETED.items(): start_ngrok_tunnel(n_ports, str(n_protocol)) prev_active_tunnels = [] while(1): active_tunnels = get_active_ngrok_tunnels() ngrok_process = ngrok.get_ngrok_process() if not ngrok_process.healthy(): break # do reconnect if if_tunnel_changes(active_tunnels, prev_active_tunnels): _Log__("INFO", active_tunnels) write_public_urls_to_file(active_tunnels) update_github_with_url_changes(url_change_indx, reconnect_indx, is_reboot) prev_active_tunnels = active_tunnels url_change_indx = url_change_indx + 1 time.sleep(60) # sleep for 1 min reconnect_indx = reconnect_indx + 1 |
Source code for script_utils.py file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
''' Filename: script_utils.py ''' import subprocess def check_network_connection(): try: cmd_resp = subprocess.check_output(['curl', '-Is', 'http://www.google.com']) cmd_resp = cmd_resp.decode("utf-8") cmd_resp_list = cmd_resp.split("\r\n") header_n1 = cmd_resp_list[0] header_n1 = header_n1.strip() if header_n1 == "HTTP/1.1 200 OK": return 1 else: return 0 except: return -1 # Test functions if __name__ == "__main__": check_network_connection() |
The above script does the following:
ngrok account using an auth token (provided by ngrok while
account creation) which can placed in a file at ./auth_token.txtPORTS_TARGETED dictionaryngrok process to check if they are healthy