Never thought I would be writing about how to utilize IPv6 in 2017 because of all the excellent material on the Internet; however, I have discovered a few things:

  1. There are still technologies which have horrible support for IPv6 (including new stuff)
  2. There are people still resistant to implementing it
  3. There is material on the Internet which shows up early in Google searches which references deprecated standards

Without any further delay, I am going to outline a few items you should keep in mind when deploying your IPv6 network:

Subnet mask size

In IPv6, barring a few exceptions like point-to-point links, you should always utilize a /64 for each deployed subnet. Why? Well, if you wanted to use DHCPv6 you’ll find Microsoft’s implementation won’t even allow you to change from a /64 and even a DHCPv6 server in Linux, while it will actually run with a mask larger than a /64, it will only hand out a /64. Also, you’ll find the use of anything larger than a /64 breaks a lot of the auto-discovery mechanisms in the switch/router, namely around EUI-64, and just doesn’t make sense.

What subnet size should I get from ISP/provider/administrator?

If you’re not going to “own” your IPv6 network, that is you’re not getting an assignment with an ASN to advertise, you’re either looking to obtain a public block of addresses for use and/or you’re internal and need your networking administrators to assign you a prefix which you can further subnet yourself. There is a standard most follow to assign prefixes to “customers”.

An ISP, for instance, may have numerous /32’s (or maybe a bit larger) assigned to them for their use to distribute to customers. Lets call them ISP and you work for “company” and you’re an internal IT organization within “company” who uses “ISP”. Your company would request from the ISP an IPv6 block assignment. From one of the ISP’s /32’s you’ll get, lets say, a /48 just for the hell of it. This is how your company can break it down internally for assignment:

  • 65,536 =  /64’s
  • 32,768 = /63’s
  • 16,384 = /62’s
  • 8192 = /61’s
  • 4096 = /60’s
  • 2048 = /59’s
  • 1024 = /58’s
  • 512 = /57’s
  • 256 = /56’s
  • 128 = /55’s
  • 64 = /54’s
  • 32 = /53’s
  • 16 = /52’s
  • 8 = /51’s
  • 4 = /50’s
  • 2 = /49’s

How your company doles these out, is up to them. However, almost no one is going to just directly carve out /64’s from the assigned /48 block, that is stupid. Generally, you’re looking to summarize and aggregate where possible throughout your network and we’ll assume you’re in location “A” at “company”.

We’ll go ahead and assume the company has decided each location is assigned a /58, which gives each location a total of 64 available /64’s to use. As you see, no different than standard IPv4 in the sense of ensuring proper aggregation, except now you’re no longer having to worry about the size of a VLAN’s subnet mask, you’ll always use /64.

What about private IPv6 address space?

If you do not want a Globally Unique IPv6 address you can indeed have what is called a “Unique Local IPv6 address = ULA”. There is a guide on how to properly generate these addresses, which includes a variable which references the time and date, along with other factors to ensure absolute uniqueness.

Why does this matter with private address space? Have you ever been involved with a merger/acquisition, or having to aggregate two offices together which use the same private IPv4 subnet range? I need not say anymore because this can be a PITA! Thus, ULA, when done right, ensures this will never happen; however, there is absolutely nothing stopping you from selecting your own, basic, prefix.

IPv6 ULA uses the FC00::/7 prefix, divided into two groups:

  1. fc00::/8 – The idea for this prefix is to be administered by some authority, but no one can agree to it, so just forget about it
  2. fd00::/8 – Is defined for the generation of /48 prefixes only, using the last 40 bits to generate a random, unique, prefix, according to the algorithm in RFC4193

You will want to use option 2 and you can use online generation tools like those from SiXXs or use a tool from another resource, either way, make sure it generates a proper /48 prefix for you and is, by some degree, RFC4193 compliant.

Finally, your company’s IT department is likely to have this /48 already and is almost very likely to have assigned you a prefix according to the same standards for which they’ll dole out their Globally Unique IPv6 addresses; thus, no additional explanation needed.

Get your DNS infrastructure setup for IPv6 AAAA and PTR-record resolution

I won’t delve into this much more other than you absolutely must make sure your DNS infrastructure is setup for IPv6 AAAA-record and IPv6 PTR-record solution or you WILL have issues!

One area to ponder is the hostnames that’ll resolve when you’re in a dual-stack environment. Do you want the same hostname to return on both a A-record and AAAA-record? Well, some say no, some say yes. Me? I say you should discuss this with your vendor to ensure their solution doesn’t have a problem with this, especially in a dual-stack environment. I was told, by co-workers who know more about Vmware vCenter than I do right now, this is a problem and the returned hostnames must be different when using dual-stack based environments.

Always research and question IPv6 support on your devices

This goes for hardware and software vendors, many have made claims their stuff works with IPv6; however, what, if any, testing was done isn’t known and there are a variety of scenarios to consider. For instance:

  • Does it support native IPv6 from installation-to-operation?
  • Does it support dual-stack, from installation-to-operation?
  • How does it handle DNS requests in dual stack?
    • Does the system start with IPv6 AAAA requests and then fails over to IPv4 A-record requests?
    • If so, what is the timeout if a AAAA record is not available and it must try for an IPv4 A-record?
    • Is the order of DNS resolution preference configurable? (Can you choose to have IPv4 A-records first?)
  • What forms of address configuration are available for IPv6? (SLAAC, static, DHCPv6?)
  • What IPv6 address types are supported? (Globally Unique and/or ULA?)
  • Are there specific “sections” of configuration which cannot support IPv6?
    • For instance, in Cisco NX OS, you cannot reference an IPv6 address for use on a vPC peer keep-alive link.

More questions will come to mind, but these are from experience and I can promise you are a lot of reasons why most IPv6 implementations in the enterprise, and data center, fail. Question all vendors!

This is it for now, hope this clears up some stuff for you out there who’re thinking about their IPv6 implementation

DNSMASQ is both a DNS and DHCP server that is quick and efficient to run on Linux systems and is likely already running on your Linux box. If you’re in need of a quick DHCP server to run your environment to serve multiple DHCP scopes for different subnets in your VLAN, of which we all know the best practice is subnet == VLAN == Broadcast domain, then DNSMASQ is your go to guy and I prefer it over the ISC DHCPD server. This quick tutorial will go over the basics of how to get this setup and running and assumes you’re not going to utilize the DNS service.

Create a directory for your DHCP leases file:

sudo mkdir /opt/dnsmasq

Setup dnsmasq.conf:

#Disable the DNS server
#Setup the server to be your authoritative DHCP server
#Set the DHCP server to hand addresses sequentially
#Enable more detailed logging for DHCP
#Set your DHCP leases file location
#Create different dhcp scopes for each of the three simulated subnets here, using tags for ID
#Format is: dhcp-range=<your_tag_here>,<start_of_scope>,<end_of_scope>,<subnet_mask>,<lease_time>
#Setup different options for each of the unique subnets, since default gateways will be different
#The format for this is: dhcp-options=<your_tags_here>,<option>,<option_value> - 3 is router

Once this is complete, enable your DHCP service to start automatically. You should also check your systems firewall/IPTABLES service(s) to ensure you have created rules to allow UDP traffic over port 67 and port 68, or you can just flush your IPTABLES and/or disable your firewall, your choice, this isn't a security blog so I'll leave the choice to you, the person who knows their environment better.

First, allow me to say these indeed to do exist, the RJ-45 based 10GBaseT SFP+ modules, a company called Methode Electronics manufactures both a SFP+ based module and a X2-RJ-45; however, we’ll only really talk about why a RJ-45 based 10GBaseT SFP+ transceiver still isn’t practical for lengths beyond 30m, with present technology.

The issues
The number one issue we have, with the current technology today in 2017, is the number of transceivers required for distances greater than 30m using 10GBaseT SFP+ modules. The incredible number of transistors will consume an enormous amount of energy per port and the heat generated by the operation of such modules will be monumental, to say the least. Also, with distances greater than 30m, the amount of heat generated needs to be pulled away from the circuitry and will require large heat sinks, which will increase the bulk of the switch itself or careful consideration of airflow characteristics around the SFP+ ports, including higher speed and higher volume fans (which in turn would also consume more energy themselves) further increases the power demands of a switch utilizing SFP+ modules for 10GBaseT SFP+ modules. X2 modules are indeed out there, but X2 is a different form factor to begin with and I won’t be discussing this here.

Why do I reference 30 meters?
Why do I reference distances greater than 30 meters (30m)? Two reasons: 1. When people want to look at Cat6a/7 for long haul connectivity (to somewhat come close to the distance of multi-mode fiber optics on OM4 fiber cables) 2. Current technology at the time of this writing actually permits us to engineer a 10GBaseT SFP+ module for distances of up to 30m using about 2.5W of energy per port. Once again, please look up the company Methode Electronics and their white paper on 10GBaseT SFP+ optics, its pretty cool stuff.

Who wants this?
Now, what audience cares about utilizing such stuff as copper for distances at 100m? In the enterprise market you’ll likely never see anyone think about using copper for spanning distances close to 100m, especially in the Data Center where the copper cross-connect is disappearing in favor of 10/25/40/50/100G fiber cross-connects, because the cost of these optics are dropping fast. When I say 40G here, I am also assuming the use of Cisco 40G BiDi transceivers because they allow you to utilize existing LC based fiber infrastructure. However, service providers are still interested in utilizing copper back haul connections for distances for at 100 meters because, if the SFP+ modules are cheap enough along with the cost of laying the Copper, they’ll want to utilize this. You’ll likely see such things as connections at last mile (rather under a mile, a lot) or between two offices or central offices. Once again, price usually always wins; thus, time will tell. So, now you know, why you’re just not seeing mass produced 10GBaseT SFP+ modules on the market.

If you’re looking to use command line variables for scripting stuff you have some predefined variables in the NX-OS environment to use and you can also create your own. For now, I’ll just show you how to use the most common, the switches hostname. In some environments you’ll have to save the output of a show tech file and later on upload it via SCP. However, if you’re doing this to 2 or more switches, you’ll need unique file names to make your life easier. Instead of going to each one, you can just use the variable SWITCHNAME in the file. So, if you’re using a script or something like cluster-ssh, this makes your job easier.

sh tech all > bootflash:///shtech-$(SWITCHNAME)

I realize there is still some confusion regarding Cisco Nexus FEX as it relates to ToR connected FEX, which is a Cisco Nexus 2K FEX with a Cisco Nexus 5K/7K/9K as a parent switch, and the FEX you find in UCS, which we can refer to as “Blade-FEX”. I am going to outline what ToR (Top of Rack) FEX in this blog post, not Blade-FEX, to help bring some clarity around this still confusing terminology. This is also not meant to bring any additional ambiguity, but it is true you can use certain Cisco Nexus 22XX ToR-FEX and “parent” them to a Cisco UCS Fabric Interconnect; however, I would not classify this as Blade-FEX or ToR-FEX, I’d like to coin it with the term “Fabric-FEX”, you owe me $1.00 every time you use this, send it via paypal :). Thus, moving forward, we’re going to refer to a FEX which parents to a Cisco Nexus switch as a ToR-FEX.

Cisco Nexus FEX works thanks to the Cisco pioneered 802.1BR, click here for more information. Now, you don’t have to worry about configuring the gory details of what is essentially VN-TAG because this is all handled with a few simple commands to get your FEX up and running; however, this is just here do you know how FEX works to communicate with the parent switch underneath the sheets.

The logical representation of FEX is broken down like this:

  • Logical Interfaces (LIF) – This is simple, its the Eth1xx/1/X representation on the switch
  • Network Interfaces (NIF) – These are the physical uplinks connecting the FEX to parent, carrying the VN-TAG
  • Virtual Interface (VIF) – This is the logical interface which correlates, in software, to the physical host interface. We we wil discuss this in a minute about why this makes FEX capable of full swap of a failed FEX without reconfiguring the host ports
  • Host Interface (HIF) – These are the physical ports on the FEX which you connect your hosts to. The parent switch assigned each HIF a unique VN-TAG ID, which is roughly correlated to the above Virtual Interface (VIF) assignment.

Here is some output to take a peek at, taken from a Cisco Nexus 9332PQ switch with 2348TQ and 2348UPQ FEX attached:

slot:36, fab_if:160001f4, p_ind:f4010016, p_numelem:1
dev_inst:0, nif_no:16, hif_no:40, nif_ind:160001f4, hif_ind:1f670a00
Eth104/1/42 0x1f670a40 Down Po501 Po501 NoConf

Take notice, this is Logical port: Eth104/1/42 and there is a plethora of information regarding the port, including the HIF numer and the hif_ind. I haven’t referenced anything with Cisco as of yet, but I would believe the HIF no is the unique number assigned to the port, perhaps the VIF, and the HIF_IND may be an index ID, but I’ll investigate later. For now, just take notice that: Eth[101-199]/1/[1-48] is the LIF, which is attached to a VIF, which correlates to the HIF on the FEX. Because FEX attaches the configuration to a VIF, which is also correlated to the FEX ID, you can have your FEX member, say FEX 104, fail completely and all you need to do is just replace the failed FEX, cable it the same way and when the FEX image is downloaded it’ll reboot and continue operation without the need to rebuild the configurations.

Now, you MUST be diligent in understanding the valid UPLINK topology you can configure your ToR-FEX for, in relation to your parent switch. Always review the configuration guide for your specific model of FEX and parent switch to obtain the valid topology. In my scenarios with the Cisco Nexus 9K switches I do a single-homed, host vPC port-channel uplink topology because we can’t do a more elaborate e-vPC design with the 9K switches and our hosts will be attached with port-channels in an active-active scenario.

Finally, the configuration is simple; however, some Cisco documentation is confusing because the wording in some documents states the UPLINK port-channel is LACP enabled; thus, you would assume you configure your UPLINK as an active LACP member. This is wrong, in fact, the best method, at least from my experience with the 9K switches, is to create the port-channel you’ll be using for the UPLINK, no-shut the interface and nothing more, then move into the physical interfaces that’ll be part of this port-channel, no shut the interfaces and just assign them to the port-channel as static mode. Then, move back into the port-channel configuration mode and build your configuration. Below is the basic configuration you need to get your FEX attached to your 9K switch:

interface po500
no shut
interface eth1/21-24
channel-group 500
no shut
int po500
switchport mode fex-fabric
fex associate
no shut -

A note about setting Jumbo frames on those FEX ports. The FEX host ports will assume the maximum MTU based on the UPLINK port-channels MTU assignment. In our environments we aim to have jumbo frames end-to-end and leave it up to the specific host/OS/application to decide on its optimal packet size. Thus, if you set your MTU on the UPLINK port-channel to 2000, your MTU on will be 2000 on your host interface ports on the FEX.

As an update, and to summarize, here are the FEX types to help clear up confusion, these are not “official” terms, but these will help to clear up confusion, I hope:

  • ToR-FEX: A Nexus 2K FEX attached (Parented) to a Nexus 3K/5K/7K/9K switch for extending ports
  • Blade FEX: These modules are installed into a Cisco UCS chassis
  • Fabric-Interconect FEX: These are the same Nexus 2K FEX used for parenting to 3K/5K/7K/9K; however, now you can attach (Parent) these Nexus 2K FEX to Cisco Fabric Interconnects for the purpose of extending ports for your Fabric Interconnects, or providing a different type/speed of port.

There has been some slight confusion and ambiguity around the “single-connection” configuration statement provided by Cisco switches and routers, including SAN MDS switches. As of this writing, Cisco Nexus 9000 NXOS switches on 7.0.3.I5.1 code do not support single-connection in their tacacs host configuration; however, certain MDS switches do. In either case, if you do find yourself wondering here for the answer, let me elaborate for you.

The purpose of single-connection is to multiplex all of your TACACS authentication requests using a single TCP oriented connection from the switch to the TACACS server. Using tac_plus, an open source TACACS server, you can absolutely set the single-connection bit from say, a Cisco 9706 MDS switch; however, upon packet analysis of any TACACS authentication requests you may discover the single-connection bit is set to 0.

Refer to draft-grant-tacacs-02 and scroll to the FLAGS section for an explanation of where you will, and should, see the single-connection bit set in the TACACS flag. Basically, you’ll only ever find the bit set in the initial setup of the connection so both the TACACS server and the client agree on single-connection TCP. Thus, instead of each and every TACACS request coming through as a unique TCP connection (essentially having to use multiple sockets, sockets being the 4-tuple of SRC IP, DST IP, SRC port, and DST port) the TACACS query and response messages are just carried over the single TCP connection.

If your system supports this, its worth attempting to see if it works as it can save some resources; however, your mileage may vary.

If you have upgraded your Cisco Nexus switches to code level 7.0(3)I2(1) or higher and had flowcontrol enabled on an interface, you’ll likely find you’re not able to do a “no flowcontrol receive on” because the command was deprecated. Current recommendation is to default the switch configuration but I have a solution you can implement one switch at-a-time with a single reload to fix this issue:

copy run startup-config
copy startup-config <tftp: | scp:>
sh run | sed 's/flowcontrol receive on//g' >> bootflash:///no-flow-control-startup-config
copy bootflash:///no-flow-control-startup-config startup-config
! Do not save the running-config to startup-config - just reload one switch at-a-time

So, how do I put this? Oh yeah, I spend money on my bikes and spare absolutely no expense considering your entire life depends on the operation of just two wheels and some really tiny brakes with a lot of stopping power; thus, cheap isn’t my game, I pay to play. I am no stranger to spending money with MotoMummy either, CapitalOne and my bank account can vouch for that. Please, continue to read on because I am not just someone who is upset because their part arrived in three days, instead of two… Read the rest of this entry »

I have seen a lot of people get confused about the length of their double banjo bolt required for the Brembo RCS master cylinder.

When using Goodridge stainless brake lines you’ll want to purchase either of the following to ensure exact fit:

  • Goodridge: 993-03-31SD or 993-03-31SDBK short 30mm version
  • Brembo: 06.2228.22 or 06.2228.21
  • Pro-bolt: TIBANJOD10FR
  • Proti-bolt: M10L21-OT04
  • LuckyBike: 92-800-TI
  • Washers (3 total): ID: 10.5MM – OD:14-15mm – Thickness 1mm

Spiegler imports their bolts and makes only one size with 35mm total length with 1.5-2.0mm in thread length using 1MM thick washers. This setup, using Goodridge lines, will not work because not only is the bolt length too long but the thread length is too long. Technically, you could use 2mm thick washers to reduce this down; however, no guarantee if the banjo fittings will line up to distribute fluid properly or the bolt actually threads correctly.

As a word of caution do not cut or otherwise modify the banjo bolt to fit the Brembo master cylinder! The Spiegler bolt costs around $15-$18  and the other bolts are the same price, but you can find the Brembo for $6.00 at some local distributor or a local Ducati or Aprillia dealership. While you can say it is unlikely something will happen with your brakes with using this cut bolt, should something happen you’ll find there are fingers pointing at you saying “Not an authorized modification, if the bolt didn’t fit should have not used or modified it to fit”. I don’t know about you, but when I am at the track or going down the highway, I want to know the one bolt which seals my brake lines at my $300 Brembo master cylinder is the proper fit and is not modified to fit because it was too long.

Just think about it, you paid $300, or more if you chose a different Brembo master cylinder model, and you’re willing to hack up a $15 bolt to make it fit? Buy the Brembo or Goodridge double banjo bolts to fit properly. Even better, the Pro-bolt is much nicer, offered with a Diamond Like Coating too, and is already pre-drilled to safety wire the bolt to ensure it never vibrates loose.

See my video on this very specific topology, what I’ve encountered, and the solution I found to work for me: