Monthly fullnode report for December 2018

Introduction

A full node server is necessary to receive and broadcast blockchain data. There are API-calls for most use cases:

  • receiving the newest posts for a feed of a specific user
  • show who already voted for a post
  • show the latest transfers of a user
  • receiving the operation from the last 10 blocks
  • broadcasting an operation to the blockchain (e.g. for posting, voting, …)

All these information are organized in databases inside the full node. These databases are needing a lot of RAM, which makes them very expensive. Each server unit can handle only a limited number of responses per second. When the number of calls per second is higher than the server can handle, the delay time for each response will increase until the server breaks down. This report tries to measure the current state of all available full nodes.

Over the last 30 days, a benchmark was performed every 3 hours on all available full nodes. The results of these benchmarks are stored as account update in @fullnodeupdate. The following parameters were measured:

  • Blocks per second - It was measured how many blocks could be streamed in 30 seconds.
  • Api call duration - The mean duration of receiving a vote, a comment, and an account is measured in seconds
  • Account history operation per second - It was measured how many account history operation could be streamed in 30 seconds
  • Irreversible block time delay in seconds- The time difference of the latest irreversible block that the node returns to the newest block is measured

Available full nodes

number_working_nodes.png

The number of available full nodes is declining from 11 to currently only 7. Disabled nodes are:

  • rpc.buildteam.io - post about going offline
  • rpc.steemliberator.com is currently not working (https issue?)
  • The websocket from rpc.curiesteem.com was disabled, I will switch to the https interface.

https://steemd.minnowsupportproject.org

0_api_call.png
0_history_ops.png
0_block_time_diff.png
0_blocks_per_second.png

The https://steemd.minnowsupportproject.org node has a good performance. It handles around 2563 history operation per second, 31 blocks per second and needs 0.31 seconds for an API call. The block delay is around 46 - 81 seconds.

https://anyx.io

1_block_time_diff.png
1_api_call.png
1_blocks_per_second.png
1_history_ops.png

The https://anyx.io node has a good performance. It handles around 481 history operation per second, 2 blocks per second and needs 2.14 seconds for an API call. The block delay is around 47 - 82 seconds.

https://api.steemit.com

2_api_call.png
2_block_time_diff.png
2_blocks_per_second.png
2_history_ops.png

The https://api.steemit.com node has a good performance. It handles around 780 history operation per second, 8 blocks per second and needs 1.50 seconds for an API call. The block delay is around 47 - 82 seconds.

https://api.steem.house

3_api_call.png
3_history_ops.png
3_block_time_diff.png
3_blocks_per_second.png

The https://api.steem.house node has a good performance. It handles around 305 history operation per second, 2 blocks per second and needs 2.16 seconds for an API call. The block delay is around 47 - 84 seconds.

https://rpc.steemviz.com

4_blocks_per_second.png
4_api_call.png
4_block_time_diff.png
4_history_ops.png

The https://rpc.steemviz.com node has a good performance. It handles around 0 history operation per second, 2 blocks per second and needs 5.98 seconds for an API call. The block delay is around 47 - 81 seconds.

https://appbasetest.timcliff.com

5_api_call.png
5_blocks_per_second.png
5_block_time_diff.png
5_history_ops.png

The https://appbasetest.timcliff.com node has a good performance. It handles around 87 history operation per second, 1 blocks per second and needs 12.06 seconds for an API call. The block delay is around 45 - 193 seconds.

https://steemd.privex.io

6_block_time_diff.png
6_blocks_per_second.png
6_history_ops.png
6_api_call.png

The https://steemd.privex.io node has a good performance. It handles around 83 history operation per second, 1 blocks per second and needs 11.91 seconds for an API call. The block delay is around 49 - 84 seconds.

Conclusion

Almost all websocket connection to full nodes are disabled now. One full node is not reachable (maybe https certificate problems) and one full node was shut down. This is not a big problem as all nodes are also reachable through an https connection.

The performance of all working nodes is very consistent and all full nodes show a good performance.

There is no up- or down trend in the performance visible, which is good, as it shows that the performance of the nodes is sufficient for handling the load.

Tools and Scripts

The following python script is used for generating all plots:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
import sys
from datetime import datetime, timedelta, date
import time
import io
import json
from timeit import default_timer as timer
import logging
from beem.account import Account
from beem.steem import Steem
from beem.utils import formatTimedelta,formatTimeString, addTzInfo
from beem.nodelist import NodeList
import matplotlib as mpl

import matplotlib.pyplot as plt
log = logging.getLogger(__name__)
logging.basicConfig(level=logging.CRITICAL)


if __name__ == "__main__":

node_stm = Steem(use_condenser=True)

acc = Account("fullnodeupdate", steem_instance=node_stm)
metadata_list = []
stopdate = datetime.utcnow() - timedelta(days=30)
startdate = datetime.utcnow()
end_time = addTzInfo(datetime.utcnow())
for h in acc.history_reverse(stop=stopdate, only_ops=["account_update"]):
metadata_list.append(json.loads(h["json_metadata"]))
end_time = formatTimeString(metadata_list[-1]["parameter"]["end_time"])

timestamp = []
working_nodes = []

for metadata in metadata_list:
report = metadata["report"]
first_entry = True
node_url_list = []
for r in report:
if r["node"] not in ["https://api.steemitstage.com", "https://api.steemitdev.com", "https://steemd-appbase.steemit.com", "wss://steemd-appbase.steemit.com"]:
node_url = r["node"].split("://")[1]
if node_url not in node_url_list:
node_url_list.append(node_url)
if first_entry:
first_entry = False
timestamp.append(formatTimeString(metadata["parameter"]["end_time"]))
working_nodes.append(1)
else:
working_nodes[-1] += 1



timestamp_node = {}
block_per_sec_node = {}
history_per_sec_node = {}
block_diff_seconds_node = {}
apicall_time_node = {}
block_diff_head_delay_node = {}
block_diff_diff_head_irreversible_node = {}

for metadata in metadata_list:
report = metadata["report"]
first_entry = True
for r in report:
if r["node"] not in ["https://api.steemitstage.com", "https://api.steemitdev.com", "https://steemd-appbase.steemit.com", "wss://steemd-appbase.steemit.com"]:


if r["node"] not in timestamp_node:
timestamp_node[r["node"]] = [formatTimeString(metadata["parameter"]["end_time"])]

if r["block"]["rank"] > -1:
block_per_sec_node[r["node"]] = [(r["block"]["count"] / r["block"]["time"])]
else:
block_per_sec_node[r["node"]] = [0]
if r["history"]["rank"] > -1:
history_per_sec_node[r["node"]] = [r["history"]["count"] / r["history"]["time"]]
else:
history_per_sec_node[r["node"]] = [0]
if r["apicall"]["rank"] > -1:
apicall_time_node[r["node"]] = [(r["apicall"]["time"])]
else:
apicall_time_node[r["node"]] = [-1]
if r["block_diff"]["rank"] > -1:
block_diff_head_delay_node[r["node"]] = [(r["block_diff"]["head_delay"])]
block_diff_diff_head_irreversible_node[r["node"]] = [(r["block_diff"]["diff_head_irreversible"])]
block_diff_seconds_node[r["node"]] = [(r["block_diff"]["diff_head_irreversible"] * 3 + r["block_diff"]["head_delay"])]
else:
block_diff_head_delay_node[r["node"]] = [-1]
block_diff_diff_head_irreversible_node[r["node"]] = [-1]
block_diff_seconds_node[r["node"]] = [-1]
else:
timestamp_node[r["node"]].append(formatTimeString(metadata["parameter"]["end_time"]))

if r["block"]["rank"] > -1:
block_per_sec_node[r["node"]].append(r["block"]["count"] / r["block"]["time"])
else:
block_per_sec_node[r["node"]].append(0)
if r["history"]["rank"] > -1:
history_per_sec_node[r["node"]].append(r["history"]["count"] / r["history"]["time"])
else:
history_per_sec_node[r["node"]].append(0)
if r["apicall"]["rank"] > -1:
apicall_time_node[r["node"]].append(r["apicall"]["time"])
else:
apicall_time_node[r["node"]].append(-1)
if r["block_diff"]["rank"] > -1:
block_diff_head_delay_node[r["node"]].append((r["block_diff"]["head_delay"]))
block_diff_diff_head_irreversible_node[r["node"]].append(r["block_diff"]["diff_head_irreversible"])
block_diff_seconds_node[r["node"]].append(r["block_diff"]["diff_head_irreversible"] * 3 + r["block_diff"]["head_delay"])
else:
block_diff_head_delay_node[r["node"]].append(-1)
block_diff_diff_head_irreversible_node[r["node"]].append(-1)
block_diff_seconds_node[r["node"]].append(-1)



show_plots = False
plt.figure(figsize=(12, 6))
opts = {'linestyle': '', 'marker': '.'}
plt.plot_date(timestamp, working_nodes, **opts)

plt.grid()
plt.legend()
plt.title("Number of working nodes")
plt.xlabel("Date")
plt.ylabel("Number of working nodes")
if show_plots:
plt.show()
else:
plt.savefig("number_working_nodes.png")


image_index = 0
for node in timestamp_node:
timestamp = timestamp_node[node]
block_per_sec = block_per_sec_node[node]
apicall_time = apicall_time_node[node]

history_per_sec = history_per_sec_node[node]
block_diff_seconds = block_diff_seconds_node[node]

mean_hist = sum(history_per_sec) / len(history_per_sec)
mean_block = sum(block_per_sec) / len(block_per_sec)
mean_api = sum(apicall_time) / len(apicall_time)
mean_diff = sum(block_diff_seconds) / len(block_diff_seconds)
max_diff = 20
min_diff = 80
for x in block_diff_seconds:
if x > max_diff and x < 1000:
max_diff = x
if x < min_diff and x > 0:
min_diff = x

print("The %s node has a good performance. It handles around %.0f history operation per second, %.0f blocks per second and needs %.2f seconds for an API call. The block delay is around %.0f - %.0f seconds." % (node, mean_hist, mean_block, mean_api, min_diff, max_diff))

plt.figure(figsize=(12, 6))
opts = {'linestyle': '', 'marker': '.'}
plt.plot_date(timestamp, block_per_sec, **opts)

plt.grid()
plt.legend()
plt.title("Blocks per second for %s" % node)
plt.xlabel("Date")
plt.ylabel("Blocks per second")
if show_plots:
plt.show()
else:
plt.savefig("%d_blocks_per_second.png" % image_index)

plt.figure(figsize=(12, 6))
opts = {'linestyle': '', 'marker': '.'}
plt.plot_date(timestamp, apicall_time, **opts)

plt.grid()
plt.legend()
plt.title("Api call duration for %s" % node)
plt.xlabel("Date")
plt.ylabel("Api call duration [s]")
if show_plots:
plt.show()
else:
plt.savefig("%d_api_call.png" % image_index)

plt.figure(figsize=(12, 6))
opts = {'linestyle': '', 'marker': '.'}
plt.plot_date(timestamp, history_per_sec, **opts)

plt.grid()
plt.legend()
plt.title("History Ops over time for %s" % node)
plt.xlabel("Date")
plt.ylabel("History Operation per second")
if show_plots:
plt.show()
else:
plt.savefig("%d_history_ops.png" % image_index)

plt.figure(figsize=(12, 6))
opts = {'linestyle': '', 'marker': '.'}
plt.plot_date(timestamp, block_diff_seconds, **opts)

plt.grid()
plt.legend()
plt.title("Block time difference to newest generated block for %s" % node)
plt.xlabel("Date")
plt.ylabel("Block time difference [s]")
if show_plots:
plt.show()
else:
plt.savefig("%d_block_time_diff.png" % image_index)

image_index += 1

The benchmark for each full node was repeated every 3 hours and is stored in the @fullnodeupdate account. The benchmark can be found here:
https://github.com/holgern/steem-fullnode-update/

Repository

https://github.com/steemit/steem
This analysis is using measured data from the steem full node servers.