Running Neo4j on a Windows Server 2019 Core Server

One of the things which puts people off of Windows (and this likely won’t solve it, but hey!) is that it’s seen as BLOATWARE. Needing lots of RAM just to start up. Largely, that’s due to the UI, and whilst personally, I’m ok with the trade off (I think 2GB to be able to see what you’re doing is OK), not everyone is of that opinion. Also, there’s a shed load of things that don’t need a UI. Now, I’m not going to set up a server with SSL, the correct domain access etc – as that’s way beyond the scope of this, but I will get one up and running with Neo4j.

Java as a platform works just fine on Windows now, performance wise – there’s not much in it, the biggest difference is likely the cost of licenses – i.e. Ubuntu is free, Windows Server – not so much. But you might have licenses, and indeed engineers/admins who are already experienced in Windows, and don’t really know Linux (ahem – like me).

Things to note, there are things Windows doesn’t have by default, most notably, an ‘in-powershell-editor’ sigh – I remember good old edit but apparently, that’s not been available on x64 systems for ever, so we’ll put an editor in place for that, we’ll also be using a script to download and setup the whole thing, as it’s a lot easier than manually doing it – if you’re interested, you should be able to parse the script files to see what they are doing.

I’m going to start on the principle that you have got Server 2019 (I’m using 20H2) installed and you’re at the initial ‘admin’ prompt, in ‘Command Prompt’ (shudder). I am setting this up for me to use, so I’ll be setting some defaults that I want, you may not want/nor need to do it, but I like it, so :p

1. Install SSH

I want to SSH to my server, whilst I am running it in VMWare and so can do it via remote, I’d rather just open up MobaXTerm and connect in. To do that we need SSH to be setup, so first, just to check it’s not already installed (as of 20H2, it’s not be default), start Powershell:

C:\> powershell

Now run:

Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'

You should see if it’s installed or not, if either aren’t then run the appropriate commands below to install: (for example, I only had to install the server)

Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Now that’s done, we need to start the service, and set it to run automatically:

Start-Service sshd
Set-Service -Name sshd -StartupType 'Automatic'

And we also need to make sure the Firewall is open to allow us to SSH, a rule should have been created when the server was setup:

Get-NetFirewallRule -Name *ssh*

If you don’t see the rule OpenSSH-Server-In-TCP then you need to create it using:

New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22

2. Change the computer name

We’re nearly able to SSH, and you’ve got 2 options here, 1 – just use the current computer name, which will be a randomly generated string, and you’ll get that by running:

$env:computername

So – either use that, or change the computer name (which will force a restart):

Rename-Computer -computername 'NAME FROM ABOVE' -newname 'NEW_NAME' -force -restart

3. Update PowerShell You don’t need to do this

SSH to your server (or carry on using your VM host) – I use SSH from MobaXTerm as it allows me to do Copy/Paste easier, anyhews. To update to the Core version of PowerShell first download it:

PowerShell
curl -Uri https://github.com/PowerShell/PowerShell/releases/download/v7.1.0/PowerShell-7.1.0-win-x64.msi -OutFile "PowerShell-7.1.0-win-x64.msi"

NB. Curl fans out there, in PowerShell 5 (default for Server 2019) is actually an alias for Invoke-WebRequest – but in Core, it’s actually curl.

Anyhews, once we have that, we need to quiet install, and we’re going to Wait for the installer to finish:

Start-Process msiexec.exe -Wait -ArgumentList "/package PowerShell-7.1.0-win-x64.msi /quiet ENABLE_PSREMOTING=1 REGISTER_MANIFEST=1"

4. Set PowerShell as the default

I mean, it’s 2020, why PowerShell isn’t the default normally is a mystery to me. Anyhews, we have two places to set it – one is for the VM connection (which you may not need if you never intend to connect that way – but it might be useful), the other is for the SSH users out there:

#VM Users
Set-ItemProperty -Path 'HKLM:\Software\Microsoft\Windows NT\CurrentVersion\WinLogon' -Name Shell -Value "c:\program files\powershell\7\pwsh.exe"

#SSH Users
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "c:\program files\powershell\7\pwsh.exe" -PropertyType String -Force

5. Install Micro-Editor

This is from here, and it gives us an editor inside PowerShell, which is what we want.
Install is probably too big-a-word for it, first we’ll download the release for us:

md C:\tools
PowerShell -command "& { (New-Object Net.WebClient).DownloadFile('https://github.com/zyedidia/micro/releases/download/v2.0.8/micro-2.0.8-win64.zip', 'c:\tools\micro-2.0.8-win64.zip') }"
Expand-Archive -Force "C:\tools\micro-2.0.8-win64.zip" "C:\tools\micro"

We’ll want this to allow us to edit the neo4j.conf file if we want to, and it’s nice to be able to edit locally. We should also add this to the PATH so we don’t have to know where it is all the time,

$oldpath = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path
$newpath = "$oldpath;C:\tools\micro\micro-2.0.8"
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value $newpath

You’ll now be able to run things like:

micro neo4j.conf

and edit within the SSH session. Obviously, you can change this to your editor of choice – vim etc

6. Let’s add a non-admin user:

The first line below will allow you to enter a password without the world seeing it, I mean – on your own VM for testing, you probably don’t need to do it – but good practices and all!

$password = Read-Host -AsSecureString
New-LocalUser "USERNAME" -Password $password -FullName "User Name"

7. Restart the computer

This get’s all our settings sorted.

Restart-Computer

8. Install the Neo4j server & extras

We’re going to download a script that will download all the appropriate programs (Neo4j Server, Java 11 etc) and configure things so we’re all ready to go. NB. The step where we cd into the C:\Neo4j folder IS IMPORTANT

md c:\Neo4j
md c:\Neo4j\Scripts
cd c:\Neo4j\
powershell -command "& { (New-Object Net.WebClient).DownloadFile('https://raw.githubusercontent.com/tomgeudens/practical-neo4j/master/scripts/fourohandbeyond/windows/Setup-Windows-Single.ps1', 'C:\neo4j\Scripts\Setup-Windows-Single.ps1') }"

Once we have that, we can execute it using:

.\Scripts\Setup-Windows-Single.ps1 -password neo

This will set the default neo4j user to have the password neo

A render of the console output from running the script
What you should see if you run the script

9. Change the scripts config

By default – it will allocate a heap of 2 GiB, and 1 GiB of page cache, I’ve been memory sensitive and only allocated 2GiB in total to my server, so if I run Neo4j now, it’ll use it all up and b0rk.

Which is why we have micro

micro c:\Neo4j\neo4j-enterprise-4.2.1\conf\neo4j.conf

Go all the way to the bottom of the file and change the memory settings so they fit in your server, in my case, Heap = 256M, PageCache = 256M

Quick Micro tips

  • Backspace doesn’t work, use Delete instead
  • CTRL+S saves
  • CTRL+Q quits

10. Start Neo4j

We’re going to run this in console mode initially, allowing us to see it’s all ok, so run:

.\Scripts\start.ps1
A rendering of what the start script should render when executed
What the ‘start’ script should output.

11. Allow the Firewall to have access (Admin)

Right, we’re nearing the home stretch, we need to add a Firewall rule to allow us to access the server, which we have to do as an Admin.
Obviously, for a production environment, you’d set this all up properly, in particular the Profile – here, I’ve used 'Domain' and 'Private' – which means your network connection needs to be one of those, ideally, it would be one of those.

New-NetFirewallRule -DisplayName 'Neo4j' -Profile @('Domain', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort @('7474', '7687')
The output of the New-NetFirewallRule commandlet
The output of the New-NetFirewallRule commandlet

12. Set the Network profile to be ‘Private’ (Admin)

By default the connection you are on will be Public – which means the Firewall will block connections via our new rule, so for me, I’m going to change the network to be Private as it’s my own internal network and that’s accurate.

First, I need to get the network adapter name:

Get-NetAdapter
Showing the location of the Get-NetAdapter commandlets Network adapter name
The red arrow points to the name we need

Now we have that, we need to get the current profile, which should be public:

Get-NetConnectionProfile -InterfaceAlias Ethernet0

and lastly, set it to the Private category to allow our rule to work.

Set-NetConnectionProfile -InterfaceAlias Ethernet0 -NetworkCategory Private

13. Test it’s working

On your machine, you should be able to go to http://<your_server_name>:7474/ and connect.

Neo4j with Azure Functions

Recently, I’ve had a couple of people ask me how to use Neo4j with Azure functions, and well – I’d not done it myself, but now I have – let’s get it done!

  1. Login to your Azure Portal

  2. Create  a new Resource, and search for ‘function app’:

image

  1. Select ‘Function App’ from the Market Place:

image

  1. Press ‘Create’ to actually make one:

image

  1. Fill in your details as you want them

image

I’m assuming you’re reasonably au fait with the setting here, in essence if you have a Resource Group you want to put it into (maybe something with a VNet) then go for it, in my case, I’ve just created a new instance of everything.

  1. Create the function, and wait for it to be ready. Mayhaps make a tea or coffee, have a break from the computer for a couple of mins – it’s all good!

image

  1. When it’s ready, click on it and go into the Function App itself (if it doesn’t take you there!)

  2. Create a new function:

image

  1. We want to create an HttpTrigger function in C# for this instance:

image

  1. This gives us a ‘run.csx’ file, which will have a load of default code, you can run it if you want,

image

and you’ll see an output window appear which will say:

image

Well – good – Azure Functions work, so let’s get a connection to a Neo4j instance – now – for this I’m assuming you have an IP to connect to – you can always use the free tier on GrapheneDB if you want to play around with this.

  1. Add references to a driver

We need to add a reference to a Neo4j client, in this case, I’ll show the official driver, but it will work as well with the community driver. First off, we need to add a ‘project.json’ file, so press ‘View Files’ on the left hand side –

image

Then add a file:

image

Then call it project.json – and yes it has to be that name:

image

With our new empty file, we need to paste in the nuget reference we need:

{
   "frameworks": {
     "net46":{
       "dependencies": {
         "neo4j.driver": "1.5.2"
       }
     }
    }
}

Annoyingly if you copy/paste this into the webpage, the function will add extra ‘closing’ curly braces, so just delete those.

image

If you press ‘Save and Run’ you should get the same response as before – which is good as it means that the Neo4j.Driver package has been installed, if we look at files, we’ll see the ‘project.json.lock’ file which we want to.

image

  1. Code

We want to add our connection information now, we’re going to go basic, and just return the COUNT of the nodes in our DB. First we need to add a ‘using’ statement to our code:

So add,

using Neo4j.Driver.V1;

Then replace the code in the function with:

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
     using (var driver = GraphDatabase.Driver("bolt://YOURIP:7687", AuthTokens.Basic("user", "pass")))
     {
         using (var session = driver.Session())
         {
             IRecord record = session.Run("MATCH (n) RETURN COUNT(n)").Single();
             int count = record["COUNT(n)"].As<int>();
             return req.CreateResponse(HttpStatusCode.OK, "Count: " + count);                  
         }
     }
}

Basically, we’ll create a Driver, open a session and then return a 200 with the count!

  1. Run

You can now ‘Save and Run’ and your output window should now tell you the count:

image

  1. Done

Your first function using Neo4j, Yay!

So you want to go Causal Neo4j in Azure? Sure we can do that

So you might have noticed in the Azure market place you can install an HA instance of Neo4j – Awesomeballs! But what about if you want a Causal cluster?

image

Hello Manual Operation!

Let’s start with a clean slate, typically in Azure you’ve probably got a dashboard stuffed full of other things, which can be distracting, so let’s create a new dashboard:

image

Give it a natty name:

image

Save and you now have an empty dashboard. Onwards!

To create our cluster, we’re gonna need 3 (count ‘em) 3 machines, the bare minimum for a cluster. So let’s fire up one, I’m creating a new Windows Server 2016 Datacenter machine. NB. I could be using Linux, but today I’ve gone Windows, and I’ll probably have a play with docker on them in a subsequent post…I digress.

image

At the bottom of the ‘new’ window, you’ll see a ‘deployment model’ option – choose ‘Resource Manager’

image

Then press ‘Create’ and start to fill in the basics!

image

  • Name: Important to remember what it is, I’ve optimistically gone with 01, allowing me to expand all the way up to 99 before I rue the day I didn’t choose 001.
  • User name: Important to remember how to login!
  • Resource group: I’m creating a new resource group, if you have an existing one you want to use, then go for it, but this gives me a good way to ensure all my Neo4j cluster resources are in one place.

Next, we’ve got to pick our size – I’m going with DS1_V2 (catchy) as it’s pretty much the cheapest, and well – I’m all about being cheap.

image

You should choose something appropriate for your needs, obvs. On to settings… which is the bulk of our workload.

image

I’m creating a new Virtual Network (VNet) and I’ve set the CIDR to the lowest I’m allowed to on Azure (10.0.0.0/29) which gives me 8 internal IP addresses – I only need 3, so… waste.

image

I’m leaving the public IP as it is, no need to change that, but I am changing the Network Security Group (NSG) as I intend on using the same one for each of my machines, and so having ‘01’ on the end (as is default) offends me Smile

image

Feel free to rename your diagnostics storage stuff if you want. The choice as they say – is yours.

Once you get the ‘ticks’ you are good to go:

image

It even adds it to the dashboard… awesomeballs!

image

Whilst we wait, lets add a couple of things to the dashboard, well, one thing, the Resource group, so view the resource groups (menu down the side) and press the ellipsis on the correct Resource group and Pin to the Dashboard:

image

So now I have:

image

After what seems like a lifetime – you’ll have a machine all setup and ready to go – well done you!

image

Now, as it takes a little while for these machines to be provisioned, I would recommend you provision another 2 now, the important bits to remember are:

  • Use the existing resource group:
    image
  • Use the same disk storage
  • Use the same virtual network
  • Use the same Network Security Group
    image

BTW, if you don’t you’re only giving yourself more work, as you’ll have to move them all to the right place eventually, may as well do it in one!

Whilst they are doing their thing, let’s setup Neo4j on the first machine, so let’s connect to it, firstly click on the VM and then the ‘connect’ button

image

We need two things on the machine

  1. Neo4j Enterprise
  2. Java

The simplest way I’ve found (provided your interwebs is up to it) is to Copy the file on your local machine, and Right-Click Paste onto the VM desktop – and yes – I’ve found it works way better using the mouse – sorry CLI-Guy

Once there, let’s install Java:

image

Then extract Neo4j to a comfy location, let’s say, the ‘C’ drive, (whilst we’re here… !Whaaaaat!!??? image 

an ‘A’ drive? I haven’t seen one of those for at least 10 years, if not longer).

Anyways – extracted and ready to roll:

image

UH OH

image

Did you get ‘failed’ deployments on those two new VMs? I did – so I went into each one and pressed ‘Start’ and that seemed to get them back up and running.

#badtimes

(That’s right – I just hashtagged in a blog post)

Anyways, we’ve now got the 3 machines up and I’m guessing you can rinse and repeat the setting up of Java and Neo4j on the other 2 machines. Now.

To configure the cluster!

We need the internal IPs of the machines, we can run ‘IpConfig’ on each machine, or just look at the V-Net on the portal and get it all in one go:

image

So, machine number 1… open up ‘neo4j.conf’ which you’ll find in the ‘conf’ folder of Neo4j. Ugh. Notepad – seriously – it’s 2017, couldn’t there be at least a slight  improvement in notepad by now???

I’m not messing with any of the other settings, purely the clustering stuff – in real life you would probably configure it a little bit more. So I’m setting:

  • dbms.mode
    • CORE
  • causal_clustering.initial_discovery_members
    • 10.0.0.4:5000,10.0.0.5:5000;10.0.0.6:5000

I’m also uncommenting all the defaults in the ‘Causal Clustering Configuration’ section – I rarely trust defaults. I also uncomment

  • dbms.connectors.default_listen_address

So it’s contactable externally. Once the other two are setup as well we’re done right?

HA No chance! Firewalls – that’s right in plural. Each machine has one – which needs to be set to accept the ports:

5000,6000,7000,7473,7474,7687

image

Obviously, you can choose not to do the last 3 ports and be uncontactable, or indeed choose any combo of them.

Aaaand, we need to configure the NSG:

image

I have 3 new ‘inbound’ rules – 7474 (browser), 7687 (bolt), 7000 – Raft.

Right. Let’s get this cluster up and contactable.

Log on to one of your VMs and fire up PowerShell (in admin mode)

image

First we navigate to the place we installed Neo4j (in my case c:\neo4j\neo4j-enterprise-3.1.3\bin) and then we import the Neo4j-Management module. To do this you need to have your ExecutionPolicy set appropriately. Being Lazy, I have it set to ‘Bypass’ (Set-ExecutionPolicy bypass).

Next we fire up the server in ‘console’ mode – this allows us to see what’s happening, for real situations – you’re going to install it as a service.

You’ll see the below initially:

image

and it will sit like that until the other servers are booted up. So I’ll leave you to go do that now…

Done?

Good – now, we need to wait a little while for them to negotiate amongst themselves, but after a short while (let’s say 30 secs or less) you should see:

image

Congratulations! You have a cluster!

Logon to that machine via the IP it says, and you’ll see the Neo4j Browser, login and then run

:play sysinfo

image

You could now run something like:

Create (:User {Name:’Your Name’})

And then browse to the other machines to see it all nicely replicated.