Hi v6,
your scenario goes far beyond what the average problems deal with in this forum. But on the other hand it is more challenging and hence more interesting for me to discuss it.
The situation is like [1]. There is only one ISP router (We/the ISP have no individual network for an edge router.) That means that the ISP router (its LAN side), _an_ ISP server, client computers and CPEs (with their client computers) are part of _one_ /64 network. The CPEs then get a /56 for their client computers
/64 LAN network of ISP: 2a02:188:4401:0::/64
"/64 ISP LAN network": In what follows I will simply call this network "WAN network" because it is the "WAN" from CPE's perspective.
Though called "WAN" it is nothing more than a usual LAN using Ethernet based on a L2 fiber network which is populated by the LAN interface of your "ISP router" (using pfSense), 28 CPE WAN interfaces and additional "client computers" you didn't mention so far. So this WAN network resembles a DMZ where the directly connected client computers will possibly be used as servers (Web, FTP, DNS, NTP, ...) (?) and the CPEs operate as firewalls that connect to protected networks.
(I tried to make some /127 routes, but it seems that the ISP router (that uses pfSense) do not follow/is compatible with RFC 6164 (Using 127-Bit IPv6 Prefixes on Inter-Router Links) so I gave up on that. /126 does however work, but I also gave up on that and instead chose to have a more simple setup.)
Using /127 would have made sense only in case
[2] but your scenario is case
[1] where you have more than 2 network nodes that want to talk to each other via the WAN network. This means you have to use a /64 as you did.
I have however a problem. :-/ I can neither from clients of the 860L reach clients in the /64 ISP LAN network as well as clients from the /64 ISP LAN network can neither get in contact with clients in the 860L /56 network.
...
What I wish is to setup routing _once_ and then clients and CPEs should be able to reach each others subnets, but without having to setup routes manually. I thought setting that route (mentioned above) should have been enough.
Just for clarity: Besides simple Internet access for clients behind the CPEs you want to establish the following communication relationships:
- Clients behind a CPE shall be able to initiate communication to clients in the WAN network and vice versa
- Clients behind one CPE shall be able to initiate communication to any client behind another CPE (not quite sure if I understood you right and this is a demand either)
What kind of reachability tests did you do, just PING6 (meaning ICMPv6 echoes) or also trying TCP/IPv6 connections or UDP/IPv6 requests?
Can the clients connected to your WAN network ping6 each other, the ISP router and any CPE's WAN interface?
Is a PING6 test within DIR-860L successful for any client connected to your WAN network or other CPE's WAN interface?
Perhaps I should add some details
/64 LAN network of ISP: 2a02:188:4401:0:: /64
ISP route to 2a02:188:4401:8100::/56 from gw address 2a02:188:4401:0::8100
From DIR-860L IPv6 status page and routing page:
...
Should these routes be enough for inter subnet communication and particularly between my CPE (and computers of my CPE) and ISP LAN (and computers of ISP LAN)?
If you neglect any possible negative influence of the DIR-860L IPv6 firewall (meaning it will allow any traffic in both directions -- and this might be not the case in your present configuration) these routes should be enough though not optimal, given that the clients connected to your WAN network will use your ISP router's address (2a02:188:4401::1) as default gateway (please check this) and that your ISP router has complete routing information for all /56 networks behind the CPEs.
For example:
- Client behind a CPE wants to talk to a client connected to your WAN network: IPv6 packet is sent from client to the CPE because it is the client's default gateway. CPE looks at destination address and discovers that this address belongs to the WAN network it is directly attached to. It forwards the packet directly to to the destination.
But: If the client in the WAN network wants to send back a reply to the client behind the CPE it has no adequate routing information for doing so. Hence it simply sends the reply packet to its default gateway which should be your ISP router 2a02:188:4401::1. Your ISP router should have a routing entry for the /56 the destination address of the reply packet belongs to and hence forward the packet to the corresponding CPE WAN interface. Hence the reply packet is routed back through the network it came from and that's why your ISP router will also send back an ICMPv6-Redirect packet to the WAN client this way conveying the host routing information, that the next time the WAN client wants to send a packet to the specific client behind the CPE it might directly send it to the CPE WAN interface omitting the ISP router as an interim hop. Hence the WAN client learns a host route to the specific client behind the CPE but only keeps it within its so called "destination cache" and usually forgets it within less than a minute if no further communication to this destination occurs - hence the next time it has to learn it again, that's what I meant when saying "not optimal" - Client connected to your WAN network wants to talk to a client behind a CPE: This is the same as for the reply packet in the case above (first bullet: paragraph starting with But:) while a reply packet now send back from the client behind the CPE to the client in the WAN network follows the path described by the first paragraph in the first bullet.
- Client behind a CPE talking to a client behind another CPE: Works similar: CPE forwards the packet due to lack of better information to your ISP router which in turn forwards it to the destination CPE thereby sending back an ICPMv6-Redirect to the source CPE hence it will temporarily cache a host route for the destination
I suppose depending on its present configuration the CPE firewall might block either ICMPv6 echo replies or generally any connection requests coming from outside. In order to allow connection initiation from WAN network clients and clients behind any other CPE you must allow any incoming traffic coming from 2a02:188:4401::/48.
Hence I propose to configure the following IPv6 firewall settings within your CPEs:
(1) Switch FW on via "Turn IPv6 Firewall ON and ALLOW rules listed"
(2) Specify a rule that allows anything going out:
- Activate the checkbox of the first rule
- Give it a Name, e.g. "AllowAllOut"
- Schedule = Always
- Source Interface = LAN
- Source IP Address Range = :: (*)
- Protocol = Any
- Dest Interface = WAN
- Dest IP Address Range = ::
- Port Range: leave as is, not relevant (=any value) for Protocol = Any
(*) Depending on model you might have to specify the range as START - END:
:: - ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
(3) Specify a second rule that allows packets with source address within 2a02:188:4401::/48 coming in:
- Activate the checkbox of the second rule
- Give it a Name, e.g. "AllowISPIn"
- Schedule = Always
- Source Interface = WAN
- Source IP Address Range = 2a02:188:4401::/48 (*)
- Protocol = Any
- Dest Interface = LAN
- Dest IP Address Range = ::
- Port Range: leave as is, not relevant (=any value) for Protocol = Any
(*) Depending on model you might have to specify the range as START - END:
2a02:188:4401:: - 2a02:188:4401:ffff:ffff:ffff:ffff:ffff
(4) If the "Simple Security" option is available activate it.
The pfSense LAN DHCP-PD functionality seems to be broken. It does not create routes on the fly (if it was intended to be that way) neither it seems to "link" a wan ip given to the CPE with the correct /56 network - if a route was created manually in advance by the netadministrator.
Sorry but I don't have any pfSense knowledge so I can't help you here.
PacketTracer