Network

The network is maintained by the RBG Network and System Group.

VLANs

We have five different VLANs/Networks.

VLAN-Name:il11il11_2il11_3il11_4il11_5il11_6il11_8
VLAN-ID:vlan_83vlan_132vlan_133vlan_235vlan_240vlan_241vlan_253
Note:chairserver mgmt (iDrac)server internwlan (free)server mwntestbed backtestbed front
Netzwerk:131.159.24.0/2310.0.0.0/810.200.112.0/20172.24.21.192/27172.24.24.0/2310.200.104.0/2110.200.96.0/21
Gateway:131.159.25.25410.0.0.110.200.127.254172.24.21.219(DHCP:..21.220)172.24.25.25410.200.111.25410.200.103.254
HostMin:131.159.24.110.0.0.110.200.112.1172.24.21.193172.24.24.110.200.104.110.200.96.1
HostMax:131.159.25.25310.255.255.25410.200.127.249172.24.21.222172.24.25.25310.200.111.24910.200.103.254
Broadcast:131.159.25.25510.255.255.25510.200.127.255172.24.21.223172.24.25.25510.200.111.25510.200.103.255
Netzmaske:255.255.254.0 = 23255.0.0.0 = 8255.255.240.0 = 2055.255.255.224 = 27255.255.254.0 = 23 255.255.248.0 = 21255.255.248.0 = 21
Hosts/Netz:510409440943051020462046

IP address

The addresses from this range can be directly assigned to the device. For some networks an internal RBG DHCP server can be used to automatically provide addresses.
Chair IPs (il11)
Chair addresses can be managed through the RBG Webapp. The MAC address needs to be whitelisted for the DHCP server. In this way each IP address is coupled with a user (MAC address).

DHCP Configurations

il11_3 (server)

  • 10.200.112.1 - 10.200.119.254 (0.200.112.0/21)
    • → no DHCP
  • 10.200.120.1 - 10.200.127.254 (10.200.120.0/21)
    • → static DHCP
    • → create a new entry on StrukDB and IP address will be assigned to the Host by DHCP

il11_5 (mwn)

  • 172.24.24.1 - 172.24.24.254 (172.24.24.0/24)
    • → static DHCP (StrukDB)
  • 172.24.25.1 - 172.24.25.254 (172.24.25.0/24)
    • → no DHCP

il11_6 (testbed)

  • 10.200.96.1 - 10.200.109.254
    • → no DHCP
  • 10.200.110.1 - 10.200.111.254 (10.200.110.0/23)
    • → static DHCP
    • → assign IPs only with entry in StrukDB

VPN

The VPN access is granted by an entry in a LDAP group (vpnil11). Only employees should be added to the ldap group!! The Ldap group is managed through the RBGWebapp.
To give students access to chair services/vms/servers the firewall should be opened to allow connections from the LRZ VPN range to this particular service/vm.

Firewall

The chair firewall is managed by the network group. An E-Mail has to be sent with the rule that needs to be added. Each network has its own firewall (→ ac_il11, ac_il11_2, etc.). Usually the chair firewall is meant. The specific rules and setup can be found under firewall
E.g. vmott4 (IP) ← Port 443 ← alle Netze, net01 (IP) ← Port 22 ← LRZ VPN, etc.

New Firewall Rules

Before adding any new firewall rules read the instructions under firewall!!

WLAN il11

The network group does not like external access points. They interfere with the existing wifi infrastructure and the situation gets worse. We have one hidden chair network that is accessible with only a password from all access points that also broadcast the eduroam network in the FMI building. Two ports of the meeting room switch are also connected to this network. Visitors can plug in a Ethernet cable and should get immediate internet connection.

VLAN:il11_4

look at the table above for the rest of the information
This network has a gateway to control access to our chair network, this gateway is managed by us and can be accessed on vmott3. More information on how the gateway is set up can be found in the sections below.

This is the gateway controlling the access from the chair WIFI (il11_4) to the rest of the chair network. This gateway is vmott3.

Setup

  • Uncomment net.ipv4.ip_forward=1 in /etc/sysctl.conf
  • Enable network interface in /etc/network/interfaces
    auto ens192
    auto ens192
    iface ens192 inet dhcp 
  • iptable rules ens160 = chair network, ens192 = il11 network
    #nat everything except traffic between chair and il11 (both ways)
    iptables -t nat -A POSTROUTING  ! -s 131.159.24.0/23 ! -d 131.159.24.0/23 -j SNAT --to-source 131.159.24.141
    #allow traffic only from chair network to gateway (INPUT)
    iptables -A INPUT -i ens160 -m state --state RELATED,ESTABLISHED -j ACCEPT
    iptables -A INPUT -i ens160 -s 131.159.24.0/23 -j ACCEPT
    #allow special traffic to be forwarded (FORWARD)
    iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
    # allow influxDB and MonoDB from il11 to testbed02 (FORWARD)
    iptables -A FORWARD -i ens192 -o ens160 -p tcp -s 172.24.21.192/27 -d 131.159.24.150 --sport 1024:65535 --dport 8086 -j ACCEPT
    iptables -A FORWARD -i ens192 -o ens160 -p tcp -s 172.24.21.192/27 -d 131.159.24.150 --sport 1024:65535 --dport 27017 -j ACCEPT
    # allow influx DB and Node.js from testbed02 to il11
    iptables -A FORWARD -i ens160 -o ens192 -p tcp -s 131.159.24.150 -d 172.24.21.192/27 --dport 8086 -j ACCEPT
    iptables -A FORWARD -i ens160 -o ens192 -p tcp -s 131.159.24.150 -d 172.24.21.192/27 --dport 8080 -j ACCEPT
    # allow MYSQL from vmott1 
    iptables -A FORWARD -i ens160 -o ens192 -p tcp -s 131.159.24.38 -d 172.24.21.192/27 --dport 3306 -j ACCEPT
    #set default policy to drop any traffic forwarded or addressed to the gateway machine
    iptables -P INPUT DROP
    iptables -P FORWARD DROP
  • save iptables to a file for reboot
     iptables-save > /etc/firewall.conf 

    add new file /etc/network/if-up.d/iptables with following content

    #!/bin/sh
    iptables-restore /etc/firewall.conf 

    make the file executable

     chmod +x /etc/network/if-up.d/iptables
  • save changes on your firewall with iptables-save > /etc/firewall.conf

Server Network

Servers are wired as instructed by the chair admin at installation time by the RBG-network group. There are usually multiple uplinks on one server:

  • iDrac (mandatory): One Ethernet cable goes into the floor to a RBG router, the port is “switched” to the il11_2 network where all server iDracs are connected to. This network is only reachable from vmott2, a RBG VM with two interfaces where one is connected to il11_2 and the other one to the chair network.
  • Chair (optional): One Ethernet cable goes into the floor to a RBG router, the port is “switched” to the il11 network which is our chair network. The Mac address of this port has to be declared in the RBG Webapp in order to get an IP address. After that the server is reachable from the chair network as any other machine.
  • Brocade Router (optional): One Fiber cable goes to the Brocade Switch at the top of the right Rack. There are multiple options to use this connection. The RBG controls the Brocade Switch and can put any network/vlan on any port. Each port on the switch has one default/untagged vlan and can optionally have multiple additional tagged vlans.

UPDATE:

  • Chair: One Ethernet cable goes into the floor to a RBG router, the port is “switched” to the il11_3 network, which is the server management network. The MAC address of this port (Interface 4 in iDrac) has to be declared in the RBG Webapp in order to get an IP address. After that the server is reachable from the chair network as any other machine.

The server room network consists of our own infrastructure and uplinks to the RBG network. Most of the hardware is wired internally with fibreglass (10G/40G). Most of the servers have a direct link to the RBG switches (chair-network + internet) with a normal 1Gb Ethernet cable. Each server should have at least two interfaces. One interface for the iDrac network (cm-mgmt, Server Management) and one for the chair-network/server-network.

Transceiver, Cables, etc.

Most of our network cards and all the switches have SFP+ Ports (10G) and QSFP (40G) Ports. The Port itself can be equipped with any kind of transceiver. The transceiver converts the electrical signal to the cable and on the other side back to the electrical signal. So the same port can be used for copper and also fiberglass. For each connection between server and switch two transceivers are needed and one cable in between.

SFP+ Transceiver

SFP+ Ports are for bandwidths up to 10G. If purchasing transceiver they should be chosen by compatibility between transceiver and server/switch. For the Intel X520 Network cards the E10GSFPSR Transceiver is compatible. With this number also OEM products can be chosen. They don't need to be directly from the original vendor (e.g. Intel) but Vendor certified (OEM) guarantees compatibility. We need SR transceiver for Short Lengths, there are also LR Transceiver which are used for lengths starting from a couple of kilometers. For the exact model and number you can alway ask the RBG-Network group mailto:rbg-noc@in.tum.de.

DAC - Direct Attach Cables

There are cables that already provide two transceiver and one cable in the middle. For example the cables from the emu03-emu14 servers to the Dell Switch are all Dell Networking Direct Attach Copper Cables with 3m. They are basically transceiver and cable combined.